docs/APPC Logging Guide/APPC Logging Guide.rst

.. ============LICENSE_START==========================================
.. ===================================================================
.. Copyright © 2017 AT&T Intellectual Property. All rights reserved.
.. ===================================================================
.. Licensed under the Creative Commons License, Attribution 4.0 Intl.  (the "License");
.. you may not use this documentation except in compliance with the License.
.. You may obtain a copy of the License at
.. 
..  https://creativecommons.org/licenses/by/4.0/
.. 
.. Unless required by applicable law or agreed to in writing, software
.. distributed under the License is distributed on an "AS IS" BASIS,
.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
.. See the License for the specific language governing permissions and
.. limitations under the License.
.. ============LICENSE_END============================================
.. ECOMP is a trademark and service mark of AT&T Intellectual Property.

APPC Logging Guide
==================

The APPC modules manage logging functionality according to the logging
configuration file ``org.ops4j.pax.logging.cfg``. APPC configuration defines
the location of the logging configuration file. Each APPC module
configured to generate logs reads the configuration file periodically
and generates logs according to the current logging level.

Logging Architecture 
---------------------

The following diagram illustrates the APPC logging architecture.

|image0|

Log Types
---------

APPC generates several types of logs to capture information needed to
operate, troubleshoot, and report on performance:

-  **Audit Log**: The Audit Log provides a summary view of the
   processing of a request – a transaction – within an application. It
   captures activity requests received by APPC and includes such
   information as the time the activity initiated, when it finishes, and
   the API invoked at the component.

-  **Metric Log**: The Metrics Log provides a more detailed view into
   the processing of a transaction within APP-C. It captures the
   beginning and ending of activities necessary to complete the
   transaction within APPC modules, for example, the Dispatcher module,
   and other ONAP components such as A&AI.

-  **Error Log**: The Error Log captures INFO, WARN, ERROR, and FATAL
   conditions sensed (“exception handled”) by the APPC software
   components.

-  **Debug Log**: The optional Debug Log captures data that may be required to debug and correct abnormal conditions of the application.


The APPC modules manage logging functionality according to the logging
configuration file ``org.ops4j.pax.logging.cfg``. APPC configuration defines
the location of the logging configuration file. Each APPC module
configured to generate logs reads the configuration file periodically
and generates logs according to the current logging level.

Logging Levels
~~~~~~~~~~~~~~

+-------------+----------------+---------------+-------------+-------------+------------+
| **Level**   | **Log type**                                                            |
+=============+================+===============+=============+=============+============+
|             |                                | **Errors**                             |
+-------------+----------------+---------------+-------------+-------------+------------+
|             | **Audit**      | **Metrics**   | **FATAL**   | **ERROR**   | **WARN**   |
+-------------+----------------+---------------+-------------+-------------+------------+
| OFF         | No             | No            | No          | No          | No         |
+-------------+----------------+---------------+-------------+-------------+------------+
| FATAL       | Yes            | Yes           | Yes         | No          | No         |
+-------------+----------------+---------------+-------------+-------------+------------+
| ERROR       | Yes            | Yes           | Yes         | Yes         | No         |
+-------------+----------------+---------------+-------------+-------------+------------+
| WARN        | Yes            | Yes           | Yes         | Yes         | Yes        |
+-------------+----------------+---------------+-------------+-------------+------------+

Response Codes
~~~~~~~~~~~~~~

+----------------------------+--------------------------------+
| **Response code ranges**   | **Error type**                 |
+============================+================================+
| 100-199                    | Permission errors              |
+----------------------------+--------------------------------+
| 200-299                    | Availability errors/Timeouts   |
+----------------------------+--------------------------------+
| 300-399                    | Data errors                    |
+----------------------------+--------------------------------+
| 400-499                    | Schema errors                  |
+----------------------------+--------------------------------+
| 500-599                    | Business process errors        |
+----------------------------+--------------------------------+
| 900-999                    | Unknown errors                 |
+----------------------------+--------------------------------+

Logging Output 
---------------

APPC generates logging output according to Event and Error Logging Framework(EELF)requirements in the
following format:

Error Log 
~~~~~~~~~~

+-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|     **Field**           |     **Description**                                                                                                                                                                                                                             |
+=========================+=================================================================================================================================================================================================================================================+
|     Timestamp           |     Date-time when error occurs in UTC.                                                                                                                                                                                                         |
+-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|     RequestID           |     Universally unique transaction requestID (UUID).                                                                                                                                                                                            |
+-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|     ThreadId            |     Blank                                                                                                                                                                                                                                       |
|                         |                                                                                                                                                                                                                                                 |
|                         |     This traces processing of a request over a number of threads of a single ONAP component.                                                                                                                                                    |
+-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|     ServiceName         |     Externally advertised API invoked by clients of this component, for example, "Audit", "HealthCheck".                                                                                                                                        |
+-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|     PartnerName         |     Client or user invoking the API.                                                                                                                                                                                                            |
+-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|     TargetEntity        |     ONAP component/subcomponent or non-ONAP entity invoked for this suboperation’s activities, for example, A&AI, VF, VM, MySql.                                                                                                                |
+-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|     TargetServiceName   |     Known name of External API/operation activities invoked on TargetEntity (ONAP component/subcomponent or non-ONAP entity), for example, VM UUID, VF UUID.                                                                                    |
+-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|     ErrorCategory       |     WARN or ERROR                                                                                                                                                                                                                               |
+-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|     ErrorCode           |     Code representing the error condition according to the error categories.                                                                                                                                                                    |
+-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|     ErrorDescription    |     Human readable description of the error - Error name, for example: "CONFIGURATION\_STARTED".                                                                                                                                                |
+-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|     detailMessage       |     This field may contain any additional information useful in describing the error condition, for example:                                                                                                                                    |
|                         |                                                                                                                                                                                                                                                 |
|                         |     ``Application Controller (APP-C) initialization started at {0}|\``                                                                                                                                                                          |
|                         |     ``No resolution is required, this is an informational message|\``                                                                                                                                                                           |
|                         |     ``The APP-C component configuration has been started because the component is being initialized or loaded for the first time, or a new instance of the component is being created. This message indicates that the component is starting.`` |
+-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Debug Log
~~~~~~~~~

+-----------------+------------------------------------------------------------+
|     **Field**   |     **Description**                                        |
+=================+============================================================+
| Timestamp       | Date-time of the log record.                               |
+-----------------+------------------------------------------------------------+
| RequestID       | Universally unique transaction requestID (UUID).           |
+-----------------+------------------------------------------------------------+
| DebugInfo       | Debug Information                                          |
+-----------------+------------------------------------------------------------+
| End of Record   | Designates the logical end of a multi-line debug record.   |
+-----------------+------------------------------------------------------------+

Audit Log
~~~~~~~~~

+--------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|     **Field**                        |     **Description**                                                                                                                                                                                       |
+======================================+===========================================================================================================================================================================================================+
|     BeginTimestamp                   |     Date-time of the start of a request activity.                                                                                                                                                         |
+--------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|     EndTimestamp                     |     Date-time of the end of a request activity.                                                                                                                                                           |
+--------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|     RequestID                        |     Universally unique transaction request ID (UUID).                                                                                                                                                     |
+--------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|     serviceInstanceID                |     Uniquely identifies a service instance, for example, “service graph”. The primary key, for example, in A&AI, to reference or manage the service instance as a unit.                                   |
+--------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|     threadId                         |     Empty                                                                                                                                                                                                 |
+--------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|     physical/virtual server name     |     Empty (the value added by the log files collecting agent).                                                                                                                                            |
+--------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|     serviceName                      |     Externally advertised API invoked by clients of this component, for example, "Audit", "HealthCheck".                                                                                                  |
+--------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|     PartnerName                      |     Client or user invoking the API.                                                                                                                                                                      |
+--------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|     StatusCode                       |     High-level success or failure of the request (COMPLETE or ERROR).                                                                                                                                     |
+--------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|     ResponseCode                     |     Application specific response code - LCM API error codes categorized according to the logging categories.                                                                                             |
+--------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|     ResponseDescription              |     Human readable description of the application specific response code, for example, "INVALID INPUT PARAMETER - ${detailedErrorMsg}".                                                                   |
+--------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|     instanceUUID                     |     Universally unique identifier to differentiate between multiple instances of the same (named), log writing component - the specific APPC instance UUID.                                               |
+--------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|     Category log level               |     Enum: “INFO” \| “WARN” \|”DEBUG” \| “ERROR” \| “FATAL”. Current log level for the entire APP-C.                                                                                                       |
+--------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|     Severity                         |     Blank                                                                                                                                                                                                 |
+--------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|     Server IP address                |     Blank                                                                                                                                                                                                 |
+--------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|     ElapsedTime                      |     Elapsed time to complete processing of an API or request at the granularity available to the component system. This value should be the difference between BeginTimestamp and EndTimestamp fields.    |
+--------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|     Server                           |     VM FQDN if virtualized, otherwise the host name of the logging component.                                                                                                                             |
+--------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|     ClientIPaddress                  |     Requesting remote client application’s IP address if known, otherwise empty.                                                                                                                          |
+--------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Metrics Log
~~~~~~~~~~~

+-----------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|     **Field**                                 |     **Description**                                                                                                                                                                                              |
+===============================================+==================================================================================================================================================================================================================+
| BeginTimestamp                                | Date-time when a suboperation activity is begun in UTC                                                                                                                                                           |
+-----------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| EndTimestamp                                  | Date-time when a supoperation activity is completed in UTC                                                                                                                                                       |
+-----------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| RequestID                                     | Universally unique transaction request ID (UUID)                                                                                                                                                                 |
+-----------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| serviceInstanceID                             | VMUUID, VFUUID                                                                                                                                                                                                   |
+-----------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| threadId                                      | Optional                                                                                                                                                                                                         |
+-----------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| physical/virtual server name                  | Empty if its value can be added by the log files collecting agent.                                                                                                                                               |
+-----------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| serviceName                                   | For example: "Audit", "HealthCheck" etc                                                                                                                                                                          |
+-----------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| PartnerName Client or user invoking the API   |                                                                                                                                                                                                                  |
+-----------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| TargetEntity                                  | APPC internal subcomponent, for example, MD-SAL, or external component, for example, A&AI, SSH, Netconf, invoked for this suboperation.                                                                          |
+-----------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| TargetServiceName                             | Operation activities invoked on TargetEntity e.g. A&AI GET generic- vnf                                                                                                                                          |
+-----------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| StatusCode                                    | High level success or failure of the suboperation activities (COMPLETE or ERROR)                                                                                                                                 |
+-----------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ResponseCode                                  | Specific response code returned by the suboperation activities.                                                                                                                                                  |
+-----------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ResponseDescription                           | Human readable description of the response code.                                                                                                                                                                 |
+-----------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| instanceUUID                                  | APPC instance ID.                                                                                                                                                                                                |
+-----------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Category log level                            | Enumerated values: “INFO” \| “WARN” \|”DEBUG” \| “ERROR” \| “FATAL”.                                                                                                                                             |
+-----------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Severity                                      | Empty                                                                                                                                                                                                            |
+-----------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Server IP address                             | The logging component host server’s IP address.                                                                                                                                                                  |
+-----------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ElapsedTime                                   | Elapsed time to complete processing of the sub operation activities at the granularity available to the component system.  This value should be the difference between EndTimestamp and BeginTimestamp fields.   |
+-----------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Server                                        | VM FQDN if virtualized, otherwise the host name of the logging component.                                                                                                                                        |
+-----------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ClientIP                                      | Requesting remote client application’s IP address.                                                                                                                                                               |
+-----------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| class name                                    | Optional. The name of the class that has caused the log record creation. For OO programing languages that support this concept.                                                                                  |
+-----------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Unused                                        |                                                                                                                                                                                                                  |
+-----------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ProcessKey                                    | Optional                                                                                                                                                                                                         |
+-----------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| TargetVirtualEntity                           | Empty                                                                                                                                                                                                            |
+-----------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| CustomField1                                  | Empty (specific attributes exposed by developers)                                                                                                                                                                |
+-----------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| CustomField2                                  | Empty                                                                                                                                                                                                            |
+-----------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| CustomField3                                  | Empty                                                                                                                                                                                                            |
+-----------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| CustomField4                                  | Empty                                                                                                                                                                                                            |
+-----------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| detailMessage                                 | Empty                                                                                                                                                                                                            |
+-----------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Log File Locations
------------------

The logging configuration file, ``org.ops4j.pax.logging.cfg`` are located in
appc Git repository:

``/appc/appc-common/src/main/resources/org/onap/appc/org.ops4j.pax.logging.cfg``


The logs are stored at the location defined by the appropriate appender:

``log4j.appender.error.File=${karaf.data}/log/APPC/appc-error.log``

``log4j.appender.debug.file=${karaf.data}/log/APPC/appc-debug.log``

``log4j.appender.metric.File=${karaf.data}/log/APPC/appc-metric.log``

``log4j.appender.audit.File=${karaf.data}/log/APPC/appc-audit.log``


Enabling APPC Logging 
----------------------

APPC uses Event and Error Logging Framework (EELF) for application logs.
To enable EELF logging:

1. Replace the default configuration file located at
   ``/opt/opendaylight/current/etc/org.ops4j.pax.logging.cfg``

   with the configuration file that is checked into git:

   ``/appc/appc-common/src/main/resources/org/onap/appc/org.ops4j.pax.logging.cfg``

2. Stop and restart ODL controller for the configuration changes to take
   effect.

3. Verify logging changes at the following log paths:

   -  ``/opt/opendaylight/current/data/log/eelf/karaf.log`` 
      This log contains the regular karaf.log output reformatted to use
      the EELF MDC properties and the pattern that is configured in the
      ``org.ops4j.pax.logging.cfg`` file.
   -  ``/opt/opendaylight/current/data/log/APPC/<package-name>``
      This directory contains the audit, metric, error, and debug logs that are configured in the ``org.ops4j.pax.logging.cfg`` file.


**Note:**
   ``/opt/opendaylight/current/data/log/APPC/controller`` contains the logs generated from the package ``org.openecomp.\*`` (all APPC logs)

-  Error.log: alarms –ERROR level logs and above

-  Info.log: INFO level logs only

-  Debug.log: debugging – DEBUG level and above

-  Audit – AUDIT level and above

Log Rotation
------------

Log rotation is performed after every 100 MB size limit is reached. The
log rotation interval is defined as part of the EELF framework.

.. |image0| image:: APPCLoggingArchitecturediagram.png 
   :width: 6.49097in
   :height: 3.46181in