JakobKrieg | f47eeee | 2020-08-17 21:04:35 +0200 | [diff] [blame^] | 1 | .. This work is a derivative of https://wiki.onap.org/display/DW/Running+Blueprints+Processor+Microservice+in+an+IDE |
| 2 | .. This work is licensed under a Creative Commons Attribution 4.0 |
| 3 | .. International License. http://creativecommons.org/licenses/by/4.0 |
| 4 | .. Copyright (C) 2020 Deutsche Telekom AG. |
| 5 | |
| 6 | Running Blueprints Processor Microservice in an IDE |
| 7 | ==================================================== |
| 8 | |
| 9 | Objective |
| 10 | ~~~~~~~~~~~~ |
| 11 | |
| 12 | Have the processor running locally is to use the IDE to run the code, while having the database running in a container. |
| 13 | This way, code changes can be conveniently tested and debugged. |
| 14 | |
| 15 | |
| 16 | Check out the code |
| 17 | ~~~~~~~~~~~~~~~~~~~ |
| 18 | |
| 19 | Check out the code from Gerrit: https://gerrit.onap.org/r/#/admin/projects/ccsdk/cds |
| 20 | |
| 21 | Build it locally |
| 22 | ~~~~~~~~~~~~~~~~~~ |
| 23 | |
| 24 | In the checked out directory, type |
| 25 | |
| 26 | .. code-block:: bash |
| 27 | |
| 28 | mvn clean install -DskipTests=true -Dmaven.test.skip=true -Dmaven.javadoc.skip=true -Dadditionalparam=-Xdoclint:none |
| 29 | |
| 30 | Spin up a Docker container with the database |
| 31 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 32 | |
| 33 | The Blueprints Processor project uses a database to store information about the blueprints |
| 34 | and therefore it needs to be online before attempting to run it. |
| 35 | |
| 36 | One way to create the database is by using the :file:`docker-compose.yaml` |
| 37 | file present on the distribution module. This database will require a local directory to mount a volume, before running docker-compose remember to create following directory: |
| 38 | |
| 39 | .. code-block:: bash |
| 40 | |
| 41 | mkdir -p -m 755 /opt/app/cds/mysql/data |
| 42 | |
| 43 | Navigate to the docker-compose file in the distribution module: |
| 44 | |
| 45 | .. tabs:: |
| 46 | |
| 47 | .. group-tab:: Latest |
| 48 | |
| 49 | .. code-block:: bash |
| 50 | |
| 51 | cd ms/blueprintsprocessor/application/src/main/dc |
| 52 | |
| 53 | .. group-tab:: Frankfurt |
| 54 | |
| 55 | .. code-block:: bash |
| 56 | |
| 57 | cd ms/blueprintsprocessor/application/src/main/dc |
| 58 | |
| 59 | .. group-tab:: El Alto |
| 60 | |
| 61 | .. code-block:: bash |
| 62 | |
| 63 | ms/blueprintsprocessor/distribution/src/main/dc |
| 64 | |
| 65 | .. group-tab:: Dublin |
| 66 | |
| 67 | .. code-block:: bash |
| 68 | |
| 69 | ms/blueprintsprocessor/distribution/src/main/dc |
| 70 | |
| 71 | And run docker-composer: |
| 72 | |
| 73 | .. code-block:: bash |
| 74 | |
| 75 | docker-compose up -d db |
| 76 | |
| 77 | This should spin up a container of the MariaDB image in the background. |
| 78 | To check if it has worked, this command can be used: |
| 79 | |
| 80 | .. code-block:: bash |
| 81 | |
| 82 | docker-compose logs -f |
| 83 | |
| 84 | The phrase ``mysqld: ready for connections`` indicates that the database was started correctly. |
| 85 | |
| 86 | From now on, the Docker container will be available on the computer; if it ever gets stopped, |
| 87 | it can be started again by the command: |
| 88 | |
| 89 | .. code-block:: bash |
| 90 | |
| 91 | docker start <id of mariadb container> |
| 92 | |
| 93 | |
| 94 | Set permissions on the local file system |
| 95 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 96 | |
| 97 | Blueprints processor uses the local file system for some operations and, therefore, |
| 98 | need some existing and accessible paths to run properly. |
| 99 | |
| 100 | Execute the following commands to create the needed directories, and grant access to the current user to modify them: |
| 101 | |
| 102 | .. code-block:: bash |
| 103 | |
| 104 | mkdir -p -m 755 /opt/app/onap/blueprints/archive |
| 105 | mkdir -p -m 755 /opt/app/onap/blueprints/deploy |
| 106 | mkdir -p -m 755 /opt/app/onap/scripts |
| 107 | sudo chown -R $(id -u):$(id -g) /opt/app/onap/ |
| 108 | |
| 109 | Import the project on the IDE |
| 110 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 111 | |
| 112 | For this page, the chosen IDE is IntelliJ, but the instructions present here can |
| 113 | be adapted to be used in other IDEs as well. |
| 114 | |
| 115 | Go to *File | Open* and choose the :file:`pom.xml` file of the cds directory: |
| 116 | |
| 117 | |imageImportProject| |
| 118 | |
| 119 | Sometimes it may be necessary to reimport Maven project: |
| 120 | |
| 121 | |imageReimportMaven| |
| 122 | |
| 123 | |
| 124 | Override some application properties |
| 125 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 126 | |
| 127 | After the project is compiled, a Run Configuration profile overriding some application properties |
| 128 | with custom values needs to be created, to reflect the local environment characteristics. |
| 129 | |
| 130 | .. tabs:: |
| 131 | |
| 132 | .. group-tab:: Latest |
| 133 | |
| 134 | Navigate to the main class of the Blueprints Processor, the BlueprintProcessorApplication class: |
| 135 | |
| 136 | ``ms/blueprintsprocessor/application/src/main/kotlin/org/onap/ccsdk/cds/blueprintsprocessor/BlueprintProcessorApplication.kt``. |
| 137 | |
| 138 | Right-click inside it, at any point, to load the context menu and select create |
| 139 | BlueprintProcessorApplication configuration from context: |
| 140 | |
| 141 | |imageCreateRunConfigkt| |
| 142 | |
| 143 | **The following window will open:** |
| 144 | |
| 145 | |imageRunConfigKt| |
| 146 | |
| 147 | **Add the following in the field `VM Options`:** |
| 148 | |
| 149 | .. code-block:: bash |
| 150 | :caption: **Custom values for properties** |
| 151 | |
| 152 | -Dspring.profiles.active=dev |
| 153 | |
| 154 | You can override any value from **application-dev.properties** file here. Use the following pattern: |
| 155 | |
| 156 | .. code-block:: java |
| 157 | |
| 158 | -D<application-dev.properties key>=<application-dev.properties value> |
| 159 | |
| 160 | .. group-tab:: Frankfurt |
| 161 | |
| 162 | Navigate to the main class of the Blueprints Processor, the BlueprintProcessorApplication class: |
| 163 | |
| 164 | ``ms/blueprintsprocessor/application/src/main/kotlin/org/onap/ccsdk/cds/blueprintsprocessor/BlueprintProcessorApplication.kt``. |
| 165 | |
| 166 | Right-click inside it, at any point, to load the context menu and select create |
| 167 | BlueprintProcessorApplication configuration from context: |
| 168 | |
| 169 | |imageCreateRunConfigkt| |
| 170 | |
| 171 | **The following window will open:** |
| 172 | |
| 173 | |imageRunConfigKt| |
| 174 | |
| 175 | **Add the following in the field `VM Options`:** |
| 176 | |
| 177 | .. code-block:: bash |
| 178 | :caption: **Custom values for properties** |
| 179 | |
| 180 | -Dspring.profiles.active=dev |
| 181 | |
| 182 | You can override any value from **application-dev.properties** file here. Use the following pattern: |
| 183 | |
| 184 | .. code-block:: java |
| 185 | |
| 186 | -D<application-dev.properties key>=<application-dev.properties value> |
| 187 | |
| 188 | .. group-tab:: El Alto |
| 189 | |
| 190 | Navigate to the main class of the Blueprints Processor, the BlueprintProcessorApplication class: |
| 191 | |
| 192 | ``ms/blueprintsprocessor/application/src/main/java/org/onap/ccsdk/cds/blueprintsprocessor/BlueprintProcessorApplication.java.`` |
| 193 | |
| 194 | Right-click inside it, at any point, to load the context menu and select create |
| 195 | BlueprintProcessorApplication configuration from context: |
| 196 | |
| 197 | |imageCreateRunConfigJava| |
| 198 | |
| 199 | **The following window will open:** |
| 200 | |
| 201 | |imageRunConfigJava| |
| 202 | |
| 203 | **Add the following in the field `VM Options`:** |
| 204 | |
| 205 | .. code-block:: bash |
| 206 | :caption: **Custom values for properties** |
| 207 | |
| 208 | -Dspring.profiles.active=dev |
| 209 | |
| 210 | You can override any value from **application-dev.properties** file here. Use the following pattern: |
| 211 | |
| 212 | .. code-block:: java |
| 213 | |
| 214 | -D<application-dev.properties key>=<application-dev.properties value> |
| 215 | |
| 216 | .. group-tab:: Dublin |
| 217 | |
| 218 | Navigate to the main class of the Blueprints Processor, the BlueprintProcessorApplication class: |
| 219 | |
| 220 | ``ms/blueprintsprocessor/application/src/main/java/org/onap/ccsdk/cds/blueprintsprocessor/BlueprintProcessorApplication.java``. |
| 221 | |
| 222 | Right-click inside it, at any point, to load the context menu and select create |
| 223 | BlueprintProcessorApplication configuration from context: |
| 224 | |
| 225 | |imageCreateRunConfigJava| |
| 226 | |
| 227 | **The following window will open:** |
| 228 | |
| 229 | |imageRunConfigJava| |
| 230 | |
| 231 | **Add the following in that field:** |
| 232 | |
| 233 | .. code-block:: java |
| 234 | :caption: **Custom values for properties** |
| 235 | |
| 236 | -DappName=ControllerBluePrints |
| 237 | -Dms_name=org.onap.ccsdk.apps.controllerblueprints |
| 238 | -DappVersion=1.0.0 |
| 239 | -Dspring.config.location=opt/app/onap/config/ |
| 240 | -Dspring.datasource.url=jdbc:mysql://127.0.0.1:3306/sdnctl |
| 241 | -Dspring.datasource.username=sdnctl |
| 242 | -Dspring.datasource.password=sdnctl |
| 243 | -Dcontrollerblueprints.loadInitialData=true |
| 244 | -Dblueprintsprocessor.restclient.sdncodl.url=http://localhost:8282/ |
| 245 | -Dblueprintsprocessor.db.primary.url=jdbc:mysql://localhost:3306/sdnctl |
| 246 | -Dblueprintsprocessor.db.primary.username=sdnctl |
| 247 | -Dblueprintsprocessor.db.primary.password=sdnctl |
| 248 | -Dblueprintsprocessor.db.primary.driverClassName=org.mariadb.jdbc.Driver |
| 249 | -Dblueprintsprocessor.db.primary.hibernateHbm2ddlAuto=update |
| 250 | -Dblueprintsprocessor.db.primary.hibernateDDLAuto=none |
| 251 | -Dblueprintsprocessor.db.primary.hibernateNamingStrategy=org.hibernate.cfg.ImprovedNamingStrategy |
| 252 | -Dblueprintsprocessor.db.primary.hibernateDialect=org.hibernate.dialect.MySQL5InnoDBDialect |
| 253 | -Dblueprints.processor.functions.python.executor.executionPath=./components/scripts/python/ccsdk_blueprints |
| 254 | -Dblueprints.processor.functions.python.executor.modulePaths=./components/scripts/python/ccsdk_blueprints,./components/scripts/python/ccsdk_netconf,./components/scripts/python/ccsdk_restconf |
| 255 | -Dblueprintsprocessor.restconfEnabled=true |
| 256 | -Dblueprintsprocessor.restclient.sdncodl.type=basic-auth |
| 257 | -Dblueprintsprocessor.restclient.sdncodl.url=http://localhost:8282/ |
| 258 | -Dblueprintsprocessor.restclient.sdncodl.username=admin |
| 259 | -Dblueprintsprocessor.restclient.sdncodl.password=Kp8bJ4SXszM0WXlhak3eHlcse2gAw84vaoGGmJvUy2U |
| 260 | -Dblueprintsprocessor.grpcEnable=false |
| 261 | -Dblueprintsprocessor.grpcPort=9111 |
| 262 | -Dblueprintsprocessor.blueprintDeployPath=/opt/app/onap/blueprints/deploy |
| 263 | -Dblueprintsprocessor.blueprintArchivePath=/opt/app/onap/blueprints/archive |
| 264 | -Dblueprintsprocessor.blueprintWorkingPath=/opt/app/onap/blueprints/work |
| 265 | -Dsecurity.user.password={bcrypt}$2a$10$duaUzVUVW0YPQCSIbGEkQOXwafZGwQ/b32/Ys4R1iwSSawFgz7QNu |
| 266 | -Dsecurity.user.name=ccsdkapps |
| 267 | -Dblueprintsprocessor.messageclient.self-service-api.kafkaEnable=false |
| 268 | -Dblueprintsprocessor.messageclient.self-service-api.topic=producer.t |
| 269 | -Dblueprintsprocessor.messageclient.self-service-api.type=kafka-basic-auth |
| 270 | -Dblueprintsprocessor.messageclient.self-service-api.bootstrapServers=127.0.0.1:9092 |
| 271 | -Dblueprintsprocessor.messageclient.self-service-api.consumerTopic=receiver.t |
| 272 | -Dblueprintsprocessor.messageclient.self-service-api.groupId=receiver-id |
| 273 | -Dblueprintsprocessor.messageclient.self-service-api.clientId=default-client-id |
| 274 | -Dspring.profiles.active=dev |
| 275 | -Dblueprintsprocessor.httpPort=8080 |
| 276 | -Dserver.port=55555 |
| 277 | |
| 278 | |
| 279 | Add/replace the following in Blueprint's application-dev.properties file: |
| 280 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 281 | |
| 282 | .. code-block:: java |
| 283 | |
| 284 | blueprintsprocessor.grpcclient.remote-python.type=token-auth |
| 285 | blueprintsprocessor.grpcclient.remote-python.host=localhost |
| 286 | blueprintsprocessor.grpcclient.remote-python.port=50051 |
| 287 | blueprintsprocessor.grpcclient.remote-python.token=Basic Y2NzZGthcHBzOmNjc2RrYXBwcw== |
| 288 | |
| 289 | blueprintprocessor.remoteScriptCommand.enabled=true |
| 290 | |
| 291 | |
| 292 | Run the application |
| 293 | ~~~~~~~~~~~~~~~~~~~ |
| 294 | |
| 295 | Select either run or debug for this Run Configuration to start the Blueprints Processor: |
| 296 | |
| 297 | |imageRunDebug| |
| 298 | |
| 299 | |imageBuildLogs| |
| 300 | |
| 301 | |
| 302 | Testing the application |
| 303 | ~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 304 | |
| 305 | There are two main features of the Blueprints Processor that can be of interest of a developer: |
| 306 | blueprint upload and blueprint process. |
| 307 | |
| 308 | To upload custom blueprints, the endpoint ``api/v1/execution-service/upload`` is used. |
| 309 | |
| 310 | To process, the endpoint is ``api/v1/execution-service/process``. |
| 311 | |
| 312 | Postman is a software that can be used to send these request, and an example of |
| 313 | them is present on https://www.getpostman.com/collections/b99863b0cde7565a32fc. |
| 314 | |
| 315 | |
| 316 | Possible Fixes |
| 317 | ~~~~~~~~~~~~~~~~~~~ |
| 318 | |
| 319 | Imported packages or annotiations are not found, Run Config not available? |
| 320 | ***************************************************************************** |
| 321 | |
| 322 | 1. Rebuild with ``maven install ...`` (see above) |
| 323 | 2. Potentially change Maven home directory in Settings |
| 324 | 3. Maven reimport in IDE |
| 325 | |
| 326 | Compilation error? |
| 327 | ******************** |
| 328 | |
| 329 | * Change Java Version to 11 or 8 |
| 330 | |
| 331 | .. |imageRunConfigJava| image:: media/run_config_java.png |
| 332 | :scale: 100 % |
| 333 | :align: middle |
| 334 | |
| 335 | .. |imageRunConfigKt| image:: media/run_config_kt.png |
| 336 | :scale: 100 % |
| 337 | :align: middle |
| 338 | |
| 339 | .. |imageCreateRunConfigJava| image:: media/create_run_config_java.png |
| 340 | :scale: 100 % |
| 341 | :align: middle |
| 342 | |
| 343 | .. |imageCreateRunConfigKt| image:: media/create_run_config_kt.png |
| 344 | :scale: 100 % |
| 345 | :align: middle |
| 346 | |
| 347 | .. |imageImportProject| image:: media/import_project.png |
| 348 | :scale: 100 % |
| 349 | :align: middle |
| 350 | |
| 351 | .. |imageReimportMaven| image:: media/reimport_maven.png |
| 352 | :scale: 70 % |
| 353 | :align: middle |
| 354 | |
| 355 | .. |imageRunDebug| image:: media/run_debug.png |
| 356 | :scale: 100 % |
| 357 | :align: middle |
| 358 | |
| 359 | .. |imageRunDebug| image:: media/run_debug.png |
| 360 | :align: middle |
| 361 | :scale: 100 % |
| 362 | |
| 363 | .. |imageBuildLogs| image:: media/build_logs.png |
| 364 | :align: middle |
| 365 | :scale: 100 % |