docs/sections/services/dfc/troubleshooting.rst

.. This work is licensed under a Creative Commons Attribution 4.0 International License.
.. http://creativecommons.org/licenses/by/4.0

Troubleshooting
===============

In order to find the origin of an error, we suggest to use the logs resulting from tracing, which needs to be activated.

Using the DFC REST API
""""""""""""""""""""""
The DFC supports a REST API which includes features to facilitate troubleshooting.

One REST primitive, **status**, returns statistics and status information for the DFC processing.
Here follows an example on how to use (here curl is used, but a web-browser can also be used. If you are
logged in to a container, wget can probably be used):

``curl http://127.0.0.1:8100/status  -i -X GET``

The following features are implemented by enabling so called 'actuators' in the Springboot framework used:

**loggers** - is used to control the logging level on different loggers (so you can enabled debug tracing on a certain
logger.

**logfile** - get logged information.

**health** - get health check info, there is currently no info here. But the endpoint is enabled.

**metrics** - read metrics from the Java execution environment; such as memory consumption, number of threads, open file
descriptors etc.

Here follow some examples:
Activate debug tracing on all classes in the DFC:

    ``curl http://127.0.0.1:8100/actuator/loggers/org.onap.dcaegen2.collectors.datafile -i -X POST  -H 'Content-Type:
    application/json' -d '{"configuredLevel":"debug"}'``

Read the log file:

    ``curl http://127.0.0.1:8100/actuator/logfile  -i -X GET``

Get build information:

    ``curl http://127.0.0.1:8100/actuator/info``

Get metric from the JVM. This lists the metrics that are available:

    ``curl http://127.0.0.1:8100/actuator/metrics  -i -X GET``

To see the value of a particular metric, just add \/[nameOfTheMetric] in the end of address, for example:

    ``curl http://127.0.0.1:8100/actuator/metrics/process.cpu.usage  -i -X GET``


Certificate failure
"""""""""""""""""""

If there is an error linked to the certificate, it is possible to get information about it. A possible cause for the
error can be that the expiry date of the certificate is past.

    ``keytool -list -v -keystore dfc.jks``

The command to encode the b64 jks file to local execution is (the \*.jks.b64 is in the repo and the Dockerfile is
encoding it into .jks. So when you pull from nexus, this won't be needed, only when git-checkout and java/mvn run):

    ``base64 -d dfc.jks.b64 > dfc.jks``


Common logs due to configuration errors
"""""""""""""""""""""""""""""""""""""""

**Do not rely on exact log messages or their presence, as they are often subject to change.**



.. **Missing configuration on Consul**

.. "Exception during getting configuration from CONSUL/CONFIG_BINDING_SERVICE"


DFC uses a number of configuration parameters. You can find below the kind of reply you get if any parameter is not valid:


-Wrong trustedCaPassword:

.. code-block:: json

    org.onap.dcaegen2.collectors.datafile.tasks.FileCollector     |2019-04-24T14:05:54.494Z     |WARN     |Failed to download file: PNF0 A20000626.2315+0200-2330+0200_PNF0-0-1MB.tar.gz, reason: org.onap.dcaegen2.collectors.datafile.exceptions.DatafileTaskException: Could not open connection: java.io.IOException: Keystore was tampered with, or password was incorrect     |RequestID=A20000626.2315+0200-2330+0200_PNF0-0-1MB.tar.gz     |     |     |FileCollectorWorker-2     |
    \...     |WARN     |Failed to download file: ..., reason: org.onap.dcaegen2.collectors.datafile.exceptions.DatafileTaskException: Could not open connection: java.io.IOException: Keystore was tampered with, or password was incorrect     ...
    \...     |WARN     |Failed to download file: ..., reason: org.onap.dcaegen2.collectors.datafile.exceptions.DatafileTaskException: Could not open connection: java.io.IOException: Keystore was tampered with, or password was incorrect     ...
    \...     |WARN     |Failed to download file: ..., reason: org.onap.dcaegen2.collectors.datafile.exceptions.DatafileTaskException: Could not open connection: java.io.IOException: Keystore was tampered with, or password was incorrect     ...
    org.onap.dcaegen2.collectors.datafile.tasks.ScheduledTasks     |2019-04-24T14:06:40.609Z     |ERROR     |File fetching failed, fileData


-Wrong trustedCa:

.. code-block:: json

    org.onap.dcaegen2.collectors.datafile.tasks.FileCollector     |2019-04-24T14:11:22.584Z     |WARN     |Failed to download file: PNF0 A20000626.2315+0200-2330+0200_PNF0-0-1MB.tar.gz, reason: org.onap.dcaegen2.collectors.datafile.exceptions.DatafileTaskException: Could not open connection: java.io.FileNotFoundException: **WRONGconfig/ftp.jks**     |RequestID=A20000626.2315+0200-2330+0200_PNF0-0-1MB.tar.gz     |     |     |FileCollectorWorker-2     |
    \...     |WARN     |Failed to download file: ..., reason: org.onap.dcaegen2.collectors.datafile.exceptions.DatafileTaskException: Could not open connection: java.io.FileNotFoundException: WRONGconfig/ftp.jks     ...
    \...     |WARN     |Failed to download file: ..., reason: org.onap.dcaegen2.collectors.datafile.exceptions.DatafileTaskException: Could not open connection: java.io.FileNotFoundException: WRONGconfig/ftp.jks     ...
    \...     |WARN     |Failed to download file: ..., reason: org.onap.dcaegen2.collectors.datafile.exceptions.DatafileTaskException: Could not open connection: java.io.FileNotFoundException: WRONGconfig/ftp.jks     ...
    org.onap.dcaegen2.collectors.datafile.tasks.ScheduledTasks     |2019-04-24T14:11:58.963Z     |ERROR     |File fetching failed, fileData

-Wrong keyPassword:

.. code-block:: json

    org.onap.dcaegen2.collectors.datafile.tasks.FileCollector     |2019-04-24T14:15:40.694Z     |WARN     |Failed to download file: PNF0 A20000626.2315+0200-2330+0200_PNF0-0-1MB.tar.gz, reason: org.onap.dcaegen2.collectors.datafile.exceptions.DatafileTaskException: Could not open connection: java.io.IOException: Keystore was tampered with, or password was incorrect     |RequestID=A20000626.2315+0200-2330+0200_PNF0-0-1MB.tar.gz     |     |     |FileCollectorWorker-2     |
    \...     |WARN     |Failed to download file: ..., reason: org.onap.dcaegen2.collectors.datafile.exceptions.DatafileTaskException: Could not open connection: java.io.IOException: Keystore was tampered with, or password was incorrect     ...
    \...     |WARN     |Failed to download file: ..., reason: org.onap.dcaegen2.collectors.datafile.exceptions.DatafileTaskException: Could not open connection: java.io.IOException: Keystore was tampered with, or password was incorrect     ...
    \...     |WARN     |Failed to download file: ..., reason: org.onap.dcaegen2.collectors.datafile.exceptions.DatafileTaskException: Could not open connection: java.io.IOException: Keystore was tampered with, or password was incorrect     ...
    org.onap.dcaegen2.collectors.datafile.tasks.ScheduledTasks     |2019-04-24T14:16:08.292Z     |ERROR     |File fetching failed, fileData

-Wrong keyCert:

.. code-block:: json

    org.onap.dcaegen2.collectors.datafile.tasks.FileCollector     |2019-04-24T14:20:46.308Z     |WARN     |Failed to download file: PNF0 A20000626.2315+0200-2330+0200_PNF0-0-1MB.tar.gz, reason: org.onap.dcaegen2.collectors.datafile.exceptions.DatafileTaskException: Could not open connection: java.io.FileNotFoundException: **WRONGconfig/dfc.jks (No such file or directory)**     |RequestID=A20000626.2315+0200-2330+0200_PNF0-0-1MB.tar.gz     |     |     |FileCollectorWorker-2     |
    \...     |WARN     |Failed to download file: ..., reason: org.onap.dcaegen2.collectors.datafile.exceptions.DatafileTaskException: Could not open connection: java.io.FileNotFoundException: WRONGconfig/dfc.jks (No such file or directory)     ...
    \...     |WARN     |Failed to download file: ..., reason: org.onap.dcaegen2.collectors.datafile.exceptions.DatafileTaskException: Could not open connection: java.io.FileNotFoundException: WRONGconfig/dfc.jks (No such file or directory)     ...
    \...     |WARN     |Failed to download file: ..., reason: org.onap.dcaegen2.collectors.datafile.exceptions.DatafileTaskException: Could not open connection: java.io.FileNotFoundException: WRONGconfig/dfc.jks (No such file or directory)     ...
    org.onap.dcaegen2.collectors.datafile.tasks.ScheduledTasks     |2019-04-24T14:21:16.447Z     |ERROR     |File fetching failed, fileData

-Wrong consumer dmaapHostName:

.. code-block:: json

    org.onap.dcaegen2.collectors.datafile.tasks.ScheduledTasks     |2019-04-24T14:27:06.578Z     |ERROR     |Polling for file ready message failed, exception: java.net.UnknownHostException: **WRONGlocalhost**: Try again, config: DmaapConsumerConfiguration{consumerId=C12, consumerGroup=OpenDcae-c12, timeoutMs=-1, messageLimit=1, **dmaapHostName=WRONGlocalhost**, dmaapPortNumber=2222, dmaapTopicName=/events/unauthenticated.VES_NOTIFICATION_OUTPUT, dmaapProtocol=http, dmaapUserName=, dmaapUserPassword=, dmaapContentType=application/json, trustStorePath=change it, trustStorePasswordPath=change it, keyStorePath=change it, keyStorePasswordPath=change it, enableDmaapCertAuth=false}     |RequestID=90fe7450-0bc2-4bf6-a2f0-2aeef6f196ae     |     |     |reactor-http-epoll-3     |
    \...     |ERROR     |Polling for file ready message failed, exception: java.net.UnknownHostException: *WRONGlocalhost*, config: DmaapConsumerConfiguration{..., dmaapHostName=*WRONGlocalhost*, ...}     ...
    \...     |ERROR     |Polling for file ready message failed, exception: java.net.UnknownHostException: *WRONGlocalhost*: Try again, config: DmaapConsumerConfiguration{..., dmaapHostName=*WRONGlocalhost*, ...}     ...
    \...     |ERROR     |Polling for file ready message failed, exception: java.net.UnknownHostException: *WRONGlocalhost*: Try again, config: DmaapConsumerConfiguration{..., dmaapHostName=*WRONGlocalhost*, ...}     ...
    \...     |ERROR     |Polling for file ready message failed, exception: java.net.UnknownHostException: *WRONGlocalhost*: Try again, config: DmaapConsumerConfiguration{..., dmaapHostName=*WRONGlocalhost*, ...}     ...
    \...     |ERROR     |Polling for file ready message failed, exception: java.net.UnknownHostException: *WRONGlocalhost*: Try again, config: DmaapConsumerConfiguration{..., dmaapHostName=*WRONGlocalhost*, ...}     ...

-Wrong consumer dmaapPortNumber:

.. code-block:: json

    org.onap.dcaegen2.collectors.datafile.tasks.ScheduledTasks     |2019-04-24T14:33:35.286Z     |ERROR     |Polling for file ready message failed, exception: io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: localhost/127.0.0.1:**WRONGport**, config: DmaapConsumerConfiguration{consumerId=C12, consumerGroup=OpenDcae-c12, timeoutMs=-1, messageLimit=1, dmaapHostName=localhost, **dmaapPortNumber=WRONGport**, dmaapTopicName=/events/unauthenticated.VES_NOTIFICATION_OUTPUT, dmaapProtocol=http, dmaapUserName=, dmaapUserPassword=, dmaapContentType=application/json, trustStorePath=change it, trustStorePasswordPath=change it, keyStorePath=change it, keyStorePasswordPath=change it, enableDmaapCertAuth=false}     |RequestID=b57c68fe-84bf-442f-accd-ea821a5a321f     |     |     |reactor-http-epoll-3     |
    \...     |ERROR     |Polling for file ready message failed, exception: io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: localhost/127.0.0.1:*WRONGport*, config: DmaapConsumerConfiguration{..., dmaapPortNumber=*WRONGport*, ...}     ...
    \...     |ERROR     |Polling for file ready message failed, exception: io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: localhost/127.0.0.1:*WRONGport*, config: DmaapConsumerConfiguration{..., dmaapPortNumber=*WRONGport*, ...}     ...
    \...     |ERROR     |Polling for file ready message failed, exception: io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: localhost/127.0.0.1:*WRONGport*, config: DmaapConsumerConfiguration{..., dmaapPortNumber=*WRONGport*, ...}     ...
    \...     |ERROR     |Polling for file ready message failed, exception: io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: localhost/127.0.0.1:*WRONGport*, config: DmaapConsumerConfiguration{..., dmaapPortNumber=*WRONGport*, ...}     ...
    \...     |ERROR     |Polling for file ready message failed, exception: io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: localhost/127.0.0.1:*WRONGport*, config: DmaapConsumerConfiguration{..., dmaapPortNumber=*WRONGport*, ...}     ...

-Wrong consumer dmaapTopicName:

.. code-block:: json

    org.onap.dcaegen2.collectors.datafile.tasks.ScheduledTasks     |2019-04-24T14:38:07.097Z     |ERROR     |Polling for file ready message failed, exception: java.lang.RuntimeException: DmaaPConsumer HTTP 404 NOT_FOUND, config: DmaapConsumerConfiguration{consumerId=C12, consumerGroup=OpenDcae-c12, timeoutMs=-1, messageLimit=1, dmaapHostName=localhost, dmaapPortNumber=2222, **dmaapTopicName=/events/unauthenticated.VES_NOTIFICATION_OUTPUTWRONG**, dmaapProtocol=http, dmaapUserName=, dmaapUserPassword=, dmaapContentType=application/json, trustStorePath=change it, trustStorePasswordPath=change it, keyStorePath=change it, keyStorePasswordPath=change it, enableDmaapCertAuth=false}     |RequestID=8bd71bac-68af-494b-9518-3ab4478371cf     |     |     |reactor-http-epoll-4     |
    \...     |ERROR     |Polling for file ready message failed, exception: java.lang.RuntimeException: DmaaPConsumer HTTP 404 NOT_FOUND, config: DmaapConsumerConfiguration{..., dmaapTopicName=*/events/unauthenticated.VES_NOTIFICATION_OUTPUTWRONG*, ...}     ...
    \...     |ERROR     |Polling for file ready message failed, exception: java.lang.RuntimeException: DmaaPConsumer HTTP 404 NOT_FOUND, config: DmaapConsumerConfiguration{..., dmaapTopicName=*/events/unauthenticated.VES_NOTIFICATION_OUTPUTWRONG*, ...}     ...
    \...     |ERROR     |Polling for file ready message failed, exception: java.lang.RuntimeException: DmaaPConsumer HTTP 404 NOT_FOUND, config: DmaapConsumerConfiguration{..., dmaapTopicName=*/events/unauthenticated.VES_NOTIFICATION_OUTPUTWRONG*, ...}     ...
    \...     |ERROR     |Polling for file ready message failed, exception: java.lang.RuntimeException: DmaaPConsumer HTTP 404 NOT_FOUND, config: DmaapConsumerConfiguration{..., dmaapTopicName=*/events/unauthenticated.VES_NOTIFICATION_OUTPUTWRONG*, ...}     ...
    \...     |ERROR     |Polling for file ready message failed, exception: java.lang.RuntimeException: DmaaPConsumer HTTP 404 NOT_FOUND, config: DmaapConsumerConfiguration{..., dmaapTopicName=*/events/unauthenticated.VES_NOTIFICATION_OUTPUTWRONG*, ...}     ...

-Consumer dmaapProtocol:
Not configurable.

Missing known_hosts file
""""""""""""""""""""""""
When StrictHostKeyChecking is enabled and DFC cannot find a known_hosts file, the warning information shown below is visible in the logfile. In this case, DFC acts like StrictHostKeyChecking is disabled.

.. code-block:: bash

    org.onap.dcaegen2.collectors.datafile.ftp.SftpClient     |2020-07-24T06:32:56.010Z     
    |WARN     |StrictHostKeyChecking is enabled but environment variable KNOWN_HOSTS_FILE_PATH is not set or points to not existing file [/home/datafile/.ssh/known_hosts]  -->  falling back to StrictHostKeyChecking='no'.  

To resolve this warning, provide a known_hosts file or disable StrictHostKeyChecking, see DFC config page - :ref:`strict_host_checking_config`.