blob: 4888233ac3269b4ee3a98d1d94c9119ed7714fc5 [file] [log] [blame]
Eric Debeaude420982017-11-13 22:16:07 +00001DCAE Installation
2=================
Lusheng Jia73548f2017-11-05 19:24:05 -05003
4The below steps covers manual setup of DCAE VM’s and DCAE service
5components.
6
7VESCollector
8------------
9
10 
11
12DCAE VES Collector can be configured on VM with ubuntu-16.04 image
13(m1.small should suffice if this is only service) and 20Gb cinder
14storage
15
161. Install docker
17
Eric Debeaude420982017-11-13 22:16:07 +000018.. code-block:: bash
Lusheng Jia73548f2017-11-05 19:24:05 -050019
Eric Debeaude420982017-11-13 22:16:07 +000020 sudo apt-get update
VENKATESH KUMAR04e7dec2018-02-28 13:57:43 -050021 sudo apt install docker.io
Lusheng Jia73548f2017-11-05 19:24:05 -050022
232. Pull the latest container from onap nexus
24
Eric Debeaude420982017-11-13 22:16:07 +000025.. code-block:: bash
Lusheng Jia73548f2017-11-05 19:24:05 -050026
VENKATESH KUMAR04e7dec2018-02-28 13:57:43 -050027 sudo docker login -u docker -p docker nexus.onap.org:10001
Eric Debeaude420982017-11-13 22:16:07 +000028
VENKATESH KUMAR04e7dec2018-02-28 13:57:43 -050029 sudo docker pull nexus.onap.org:10001/onap/org.onap.dcaegen2.collectors.ves.vescollector:v1.1.0
Lusheng Jia73548f2017-11-05 19:24:05 -050030
313. Start the VESCollector with below command
32
Eric Debeaude420982017-11-13 22:16:07 +000033.. code-block:: bash
Lusheng Jia73548f2017-11-05 19:24:05 -050034
Eric Debeaude420982017-11-13 22:16:07 +000035 sudo docker run -d --name vescollector -p 8080:8080/tcp -p
36 8443:8443/tcp -P -e DMAAPHOST='<dmaap IP>'
VENKATESH KUMAR04e7dec2018-02-28 13:57:43 -050037 nexus.onap.org:10001/onap/org.onap.dcaegen2.collectors.ves.vescollector:v1.1.0
Eric Debeaude420982017-11-13 22:16:07 +000038
39.. Note:  Change the dmaaphost to required DMAAP ip. To change the
40 dmaap information for a running container,  stop the active
41 container and rerun above command changing the dmaap IP.
Lusheng Jia73548f2017-11-05 19:24:05 -050042
434. Verification
44
45i. Check logs under container /opt/app/VESCollector/logs/collector.log
46 for errors
47
48ii. If no active feed, you can simulate an event into collector via curl
49
Eric Debeaude420982017-11-13 22:16:07 +000050.. code-block:: bash
Lusheng Jia73548f2017-11-05 19:24:05 -050051
Eric Debeaude420982017-11-13 22:16:07 +000052 curl -i  -X POST -d @<sampleves> --header "Content-Type:application/json" -k http://localhost:8080/eventListener/v5
53
54.. Note: If DMAAPHOST provided is invalid, you will see exception
55 around publish on the collector.logs (collector queues and attempts
56 to resend the event hence exceptions reported will be periodic). 
Lusheng Jia73548f2017-11-05 19:24:05 -050057
58i. Below two topic configuration are pre-set into this container.  When
59 valid DMAAP instance ip was provided and VES events are received,
60 the collector will post to below topics.
61
62 Fault -
63  http://<dmaaphost>:3904/events/unauthenticated.SEC\_FAULT\_OUTPUT
64
65 Measurement
66 -http://<dmaaphost>:3904/events/unauthenticated.SEC\_MEASUREMENT\_OUTPUT
67
68VM Init
Eric Debeaude420982017-11-13 22:16:07 +000069~~~~~~~
Lusheng Jia73548f2017-11-05 19:24:05 -050070
71To address windriver server in-stability, the below **init.sh** script
72was used to start the container on VM restart.  
73
Eric Debeaude420982017-11-13 22:16:07 +000074.. code-block:: bash
75
76 #!/bin/sh
77 sudo docker ps | grep “vescollector”
78 if [ $? -ne 0 ]; then
79 sudo docker login -u docker -p docker nexus.onap.org:10001
80 sudo docker pull nexus.onap.org:10001/onap/org.onap.dcaegen2.collectors.ves.vescollector:1.1
81 sudo docker rm -f vescollector
82 echo “Collector process not running - $(date)” >> /home/ubuntu/startuplog
83 sudo docker run -d –name vescollector -p 8080:8080/tcp -p 8443:8443/tcp -P -e DMAAPHOST=‘10.12.25.96’ nexus.onap.org:10001/onap/org.onap.dcaegen2.collectors.ves.vescollector:1.1
84 else
85 echo “Collector process running - $(date)” >> /home/ubuntu/startuplog
86 fi
87
Lusheng Jia73548f2017-11-05 19:24:05 -050088
89This script was invoked via VM init script (rc.d).
90
Eric Debeaude420982017-11-13 22:16:07 +000091.. code-block:: bash
Lusheng Jia73548f2017-11-05 19:24:05 -050092
Eric Debeaude420982017-11-13 22:16:07 +000093 ln -s /home/ubuntu/init.sh /etc/init.d/init.sh
94 sudo  update-rc.d init.sh start 2
Lusheng Jia73548f2017-11-05 19:24:05 -050095
96 
97
98ThresholdCrossingAnalysis (TCA/CDAP)
99------------------------------------
100
101The platform deploys CDAP as cluster and instantiates TCA. For the
102manual setup, we will leverage the CDAP SDK docker container to deploy
103TCA instances.  To setup TCA, choose VM with ubuntu-16.04 image,
104m1.medium size and 50gb cinder volumes.
105
1061. Install docker
107
Eric Debeaude420982017-11-13 22:16:07 +0000108.. code-block:: bash
Lusheng Jia73548f2017-11-05 19:24:05 -0500109
Eric Debeaude420982017-11-13 22:16:07 +0000110 sudo apt-get update
VENKATESH KUMAR04e7dec2018-02-28 13:57:43 -0500111 sudo apt install docker.io
Lusheng Jia73548f2017-11-05 19:24:05 -0500112
1132. Pull CDAP SDK container
114
Eric Debeaude420982017-11-13 22:16:07 +0000115.. code-block:: bash
116
117 sudo docker pull caskdata/cdap-standalone:4.1.2
Lusheng Jia73548f2017-11-05 19:24:05 -0500118
1193. Deploy and run the CDAP container
120
Eric Debeaude420982017-11-13 22:16:07 +0000121.. code-block:: bash
122
123 sudo docker run -d --name cdap-sdk-2 -p 11011:11011 -p 11015:11015
124 caskdata/cdap-standalone:4.1.2
Lusheng Jia73548f2017-11-05 19:24:05 -0500125
1264. Create Namespace on CDAP application
127
Eric Debeaude420982017-11-13 22:16:07 +0000128.. code-block:: bash
Lusheng Jia73548f2017-11-05 19:24:05 -0500129
Eric Debeaude420982017-11-13 22:16:07 +0000130 curl -X PUT http://localhost:11015/v3/namespaces/cdap_tca_hi_lo
Lusheng Jia73548f2017-11-05 19:24:05 -0500131
Eric Debeaude420982017-11-13 22:16:07 +00001325. Create TCA app config file - "tca\_app\_config.json" under ~ubuntu as below
133
134.. code-block:: json
135
136 {
137 "artifact": {
138 "name": "dcae-analytics-cdap-tca",
VENKATESH KUMAR04e7dec2018-02-28 13:57:43 -0500139 "version": "2.0.1",
Eric Debeaude420982017-11-13 22:16:07 +0000140 "scope": "user"
141 },
142
143 "config": {
144 "appName": "dcae-tca",
145 "appDescription": "DCAE Analytics Threshold Crossing Alert Application",
146 "tcaVESMessageStatusTableName": "TCAVESMessageStatusTable",
147 "tcaVESMessageStatusTableTTLSeconds": 86400.0,
148 "tcaAlertsAbatementTableName": "TCAAlertsAbatementTable",
149 "tcaAlertsAbatementTableTTLSeconds": 1728000.0,
150 "tcaVESAlertsTableName": "TCAVESAlertsTable",
151 "tcaVESAlertsTableTTLSeconds": 1728000.0,
152 "thresholdCalculatorFlowletInstances": 2.0,
153 "tcaSubscriberOutputStreamName": "TCASubscriberOutputStream"
154 }
155 }
156
Lusheng Jia73548f2017-11-05 19:24:05 -0500157
1586. Create TCA app preference file under ~ubuntu as below
159
Eric Debeaude420982017-11-13 22:16:07 +0000160.. code-block:: json
Lusheng Jia73548f2017-11-05 19:24:05 -0500161
Eric Debeaude420982017-11-13 22:16:07 +0000162 {
163 "publisherContentType" : "application/json",
164 "publisherHostName" : "10.12.25.96",
165 "publisherHostPort" : "3904",
166 "publisherMaxBatchSize" : "1",
167 "publisherMaxRecoveryQueueSize" : "100000",
168 "publisherPollingInterval" : "20000",
169 "publisherProtocol" : "http",
170 "publisherTopicName" : "unauthenticated.DCAE_CL_OUTPUT",
171 "subscriberConsumerGroup" : "OpenDCAE-c1",
172 "subscriberConsumerId" : "c1",
173 "subscriberContentType" : "application/json",
174 "subscriberHostName" : "10.12.25.96",
175 "subscriberHostPort" : "3904",
176 "subscriberMessageLimit" : "-1",
177 "subscriberPollingInterval" : "20000",
178 "subscriberProtocol" : "http",
179 "subscriberTimeoutMS" : "-1",
180 "subscriberTopicName" : "unauthenticated.SEC_MEASUREMENT_OUTPUT",
181 "enableAAIEnrichment" : false,
182 "aaiEnrichmentHost" : "10.12.25.72",
183 "aaiEnrichmentPortNumber" : 8443,
184 "aaiEnrichmentProtocol" : "https",
185 "aaiEnrichmentUserName" : "DCAE",
186 "aaiEnrichmentUserPassword" : "DCAE",
187 "aaiEnrichmentIgnoreSSLCertificateErrors" : false,
188 "aaiVNFEnrichmentAPIPath" : "/aai/v11/network/generic-vnfs/generic-vnf",
189 "aaiVMEnrichmentAPIPath" : "/aai/v11/search/nodes-query",
VENKATESH KUMAR04e7dec2018-02-28 13:57:43 -0500190 "tca_policy" : "{
191 \"domain\": \"measurementsForVfScaling\",
192 \"metricsPerEventName\": [{
193 \"eventName\": \"vFirewallBroadcastPackets\",
194 \"controlLoopSchemaType\": \"VNF\",
195 \"policyScope\": \"DCAE\",
196 \"policyName\": \"DCAE.Config_tca-hi-lo\",
197 \"policyVersion\": \"v0.0.1\",
198 \"thresholds\": [{
199 \"closedLoopControlName\": \"ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a\",
200 \"version\": \"1.0.2\",
201 \"fieldPath\": \"$.event.measurementsForVfScalingFields.vNicUsageArray[*].receivedTotalPacketsDelta\",
202 \"thresholdValue\": 300,
203 \"direction\": \"LESS_OR_EQUAL\",
204 \"severity\": \"MAJOR\",
205 \"closedLoopEventStatus\": \"ONSET\"
Eric Debeaude420982017-11-13 22:16:07 +0000206 }, {
VENKATESH KUMAR04e7dec2018-02-28 13:57:43 -0500207 \"closedLoopControlName\": \"ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a\",
208 \"version\": \"1.0.2\",
209 \"fieldPath\": \"$.event.measurementsForVfScalingFields.vNicUsageArray[*].receivedTotalPacketsDelta\",
210 \"thresholdValue\": 700,
211 \"direction\": \"GREATER_OR_EQUAL\",
212 \"severity\": \"CRITICAL\",
213 \"closedLoopEventStatus\": \"ONSET\"
Eric Debeaude420982017-11-13 22:16:07 +0000214 }]
215 }, {
VENKATESH KUMAR04e7dec2018-02-28 13:57:43 -0500216 \"eventName\": \"vLoadBalancer\",
217 \"controlLoopSchemaType\": \"VM\",
218 \"policyScope\": \"DCAE\",
219 \"policyName\": \"DCAE.Config_tca-hi-lo\",
220 \"policyVersion\": \"v0.0.1\",
221 \"thresholds\": [{
222 \"closedLoopControlName\": \"ControlLoop-vDNS-6f37f56d-a87d-4b85-b6a9-cc953cf779b3\",
223 \"version\": \"1.0.2\",
224 \"fieldPath\": \"$.event.measurementsForVfScalingFields.vNicUsageArray[*].receivedTotalPacketsDelta\",
225 \"thresholdValue\": 300,
226 \"direction\": \"GREATER_OR_EQUAL\",
227 \"severity\": \"CRITICAL\",
228 \"closedLoopEventStatus\": \"ONSET\"
Eric Debeaude420982017-11-13 22:16:07 +0000229 }]
230 }, {
VENKATESH KUMAR04e7dec2018-02-28 13:57:43 -0500231 \"eventName\": \"Measurement_vGMUX\",
232 \"controlLoopSchemaType\": \"VNF\",
233 \"policyScope\": \"DCAE\",
234 \"policyName\": \"DCAE.Config_tca-hi-lo\",
235 \"policyVersion\": \"v0.0.1\",
236 \"thresholds\": [{
237 \"closedLoopControlName\": \"ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e\",
238 \"version\": \"1.0.2\",
239 \"fieldPath\": \"$.event.measurementsForVfScalingFields.additionalMeasurements[*].arrayOfFields[0].value\",
240 \"thresholdValue\": 0,
241 \"direction\": \"EQUAL\",
242 \"severity\": \"MAJOR\",
243 \"closedLoopEventStatus\": \"ABATED\"
Eric Debeaude420982017-11-13 22:16:07 +0000244 }, {
VENKATESH KUMAR04e7dec2018-02-28 13:57:43 -0500245 \"closedLoopControlName\": \"ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e\",
246 \"version\": \"1.0.2\",
247 \"fieldPath\": \"$.event.measurementsForVfScalingFields.additionalMeasurements[*].arrayOfFields[0].value\",
248 \"thresholdValue\": 0,
249 \"direction\": \"GREATER\",
250 \"severity\": \"CRITICAL\",
251 \"closedLoopEventStatus\": \"ONSET\"
Eric Debeaude420982017-11-13 22:16:07 +0000252 }]
253 }]
VENKATESH KUMAR04e7dec2018-02-28 13:57:43 -0500254}"
Eric Debeaude420982017-11-13 22:16:07 +0000255 }
Lusheng Jia73548f2017-11-05 19:24:05 -0500256
Lusheng Jia73548f2017-11-05 19:24:05 -0500257
Eric Debeaude420982017-11-13 22:16:07 +0000258.. Note: Dmaap configuration are specified on this file on
259 publisherHostName and subscriberHostName. To be changed as
260 required\*\*
261
VENKATESH KUMAR04e7dec2018-02-28 13:57:43 -05002626a. To enable AAI Enrichment in TCA for CL event, following updates should be done to TCA app preference file before proceeding
263
264.. code-block:: json
265
266 "enableAAIEnrichment" : true
267 "aaiEnrichmentHost" : <should be set to aai1_ip_addr>
268 "aaiEnrichmentIgnoreSSLCertificateErrors" : true
269
270
271
Eric Debeaude420982017-11-13 22:16:07 +00002727. Copy below script to CDAP server (this gets latest image from nexus and deploys TCA application) and execute it
273
274.. code-block:: bash
275
276 #!/bin/sh
VENKATESH KUMAR04e7dec2018-02-28 13:57:43 -0500277 TCA_JAR=dcae-analytics-cdap-tca-2.0.1.jar
Eric Debeaude420982017-11-13 22:16:07 +0000278 rm -f /home/ubuntu/$TCA_JAR
279 cd /home/ubuntu/
VENKATESH KUMAR04e7dec2018-02-28 13:57:43 -0500280 wget https://nexus.onap.org/service/local/repositories/releases/content/org/onap/dcaegen2/analytics/tca/dcae-analytics-cdap-tca/2.0.1/$TCA_JAR
Eric Debeaude420982017-11-13 22:16:07 +0000281 if [ $? -eq 0 ]; then
282 if [ -f /home/ubuntu/$TCA_JAR ]; then
283 echo “Restarting TCA CDAP application using $TCA_JAR artifact”
284 else
285 echo “ERROR: $TCA_JAR missing”
286 exit 1
287 fi
288 else
289 echo “ERROR: $TCA_JAR not found in nexus”
290 exit 1
291 fi
292 # stop programs
293 curl -X POST http://localhost:11015/v3/namespaces/cdap_tca_hi_lo/apps/dcae-tca/workers/TCADMaaPMRPublisherWorker/stop
294 curl -X POST http://localhost:11015/v3/namespaces/cdap_tca_hi_lo/apps/dcae-tca/workers/TCADMaaPMRSubscriberWorker/stop
295 curl -X POST http://localhost:11015/v3/namespaces/cdap_tca_hi_lo/apps/dcae-tca/flows/TCAVESCollectorFlow/stop
296 # delete application
297 curl -X DELETE http://localhost:11015/v3/namespaces/cdap_tca_hi_lo/apps/dcae-tca
298 # delete artifact
299 curl -X DELETE http://localhost:11015/v3/namespaces/cdap_tca_hi_lo/artifacts/dcae-analytics-cdap-tca/versions/2.0.0
300 # load artifact
301 curl -X POST –data-binary @/home/ubuntu/$TCA_JAR http://localhost:11015/v3/namespaces/cdap_tca_hi_lo/artifacts/dcae-analytics-cdap-tca
302 # create app
303 curl -X PUT -d @/home/ubuntu/tca_app_config.json http://localhost:11015/v3/namespaces/cdap_tca_hi_lo/apps/dcae-tca
304 # load preferences
305 curl -X PUT -d @/home/ubuntu/tca_app_preferences.json http://localhost:11015/v3/namespaces/cdap_tca_hi_lo/apps/dcae-tca/preferences
306 # start programs
307 curl -X POST http://localhost:11015/v3/namespaces/cdap_tca_hi_lo/apps/dcae-tca/workers/TCADMaaPMRPublisherWorker/start
308 curl -X POST http://localhost:11015/v3/namespaces/cdap_tca_hi_lo/apps/dcae-tca/workers/TCADMaaPMRSubscriberWorker/start
309 curl -X POST http://localhost:11015/v3/namespaces/cdap_tca_hi_lo/apps/dcae-tca/flows/TCAVESCollectorFlow/start
310 echo
311 # get status of programs
312 curl http://localhost:11015/v3/namespaces/cdap_tca_hi_lo/apps/dcae-tca/workers/TCADMaaPMRPublisherWorker/status
313 curl http://localhost:11015/v3/namespaces/cdap_tca_hi_lo/apps/dcae-tca/workers/TCADMaaPMRSubscriberWorker/status
314 curl http://localhost:11015/v3/namespaces/cdap_tca_hi_lo/apps/dcae-tca/flows/TCAVESCollectorFlow/status
315 echo
316
Lusheng Jia73548f2017-11-05 19:24:05 -0500317
3188. Verify TCA application and logs via CDAP GUI processes
319
320 The overall flow can be checked here
321
322TCA Configuration Change
Eric Debeaude420982017-11-13 22:16:07 +0000323~~~~~~~~~~~~~~~~~~~~~~~~
Lusheng Jia73548f2017-11-05 19:24:05 -0500324
325Typical configuration changes include changing DMAAP host and/or Policy configuration. If necessary, modify the file on step #6 and run the script noted as step #7 to redeploy TCA with updated configuration.
Lusheng Jia73548f2017-11-05 19:24:05 -0500326
327VM Init
Eric Debeaude420982017-11-13 22:16:07 +0000328~~~~~~~
Lusheng Jia73548f2017-11-05 19:24:05 -0500329
330To address windriver server in-stability, the below **init.sh** script
331was used to restart the container on VM restart.  This script was
332invoked via VM init script (rc.d).
333
Eric Debeaude420982017-11-13 22:16:07 +0000334.. code-block:: bash
Lusheng Jia73548f2017-11-05 19:24:05 -0500335
Eric Debeaude420982017-11-13 22:16:07 +0000336 #!/bin/sh
337 #docker run -d –name cdap-sdk -p 11011:11011 -p 11015:11015 caskdata/cdap-standalone:4.1.2
338 sudo docker restart cdap-sdk-2
339 sleep 30
340 # start program
341 curl -X POST http://localhost:11015/v3/namespaces/cdap_tca_hi_lo/apps/dcae-tca/workers/TCADMaaPMRPublisherWorker/start
342 curl -X POST http://localhost:11015/v3/namespaces/cdap_tca_hi_lo/apps/dcae-tca/workers/TCADMaaPMRSubscriberWorker/start
343 curl -X POST http://localhost:11015/v3/namespaces/cdap_tca_hi_lo/apps/dcae-tca/flows/TCAVESCollectorFlow/start
344
Lusheng Jia73548f2017-11-05 19:24:05 -0500345
346This script was invoked via VM init script (rc.d).
347
Eric Debeaude420982017-11-13 22:16:07 +0000348.. code-block:: bash
Lusheng Jia73548f2017-11-05 19:24:05 -0500349
Eric Debeaude420982017-11-13 22:16:07 +0000350 ln -s /home/ubuntu/init.sh /etc/init.d/init.sh
351 sudo  update-rc.d init.sh start 2