| .. This work is licensed under a Creative Commons Attribution 4.0 International License. |
| .. http://creativecommons.org/licenses/by/4.0 |
| |
| |
| Cloudify DMaaP Plugin |
| ===================== |
| |
| Cloudify plugin for creating and managing DMaaP Data Router feeds and |
| subscriptions and DMaaP Message Router topics. The plugin uses the DMaaP |
| Bus Controller API. |
| |
| Plugin Support for DMaaP Data Router |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| |
| Plugin Types for DMaaP Data Router |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| |
| The Cloudify type definitions for DMaaP Data Router nodes and |
| relationships are defined in ```dmaap.yaml`` <./dmaap.yaml>`__. |
| |
| There are four node types for DMaaP Data Router: |
| |
| - ``ccsdk.nodes.Feed``: This type represents a feed that does not yet |
| exist and that should be created when the install workflow is run |
| against a blueprint that contains a node of this type. |
| |
| +------------------------+----------+-------------+---------------------------------------------------------------------------------------------+ |
| | Property | Type | Required? | Description | |
| +========================+==========+=============+=============================================================================================+ |
| | feed\_name | string | no | a name that identifies the feed (plugin will generate if absent) | |
| +------------------------+----------+-------------+---------------------------------------------------------------------------------------------+ |
| | feed\_version | string | no | version number for the feed (feed\_name + feed\_version uniquely identify the feed in DR) | |
| +------------------------+----------+-------------+---------------------------------------------------------------------------------------------+ |
| | feed\_description | string | no | human-readable description of the feed | |
| +------------------------+----------+-------------+---------------------------------------------------------------------------------------------+ |
| | aspr\_classification | string | no | AT&T ASPR classification of the feed | |
| +------------------------+----------+-------------+---------------------------------------------------------------------------------------------+ |
| |
| - ``ccsdk.nodes.ExistingFeed``: This type represents a feed that |
| already exists. Nodes of this type are placed in a blueprint so that |
| other nodes in the blueprint can be set up as publishers or |
| subscribers to the feed. The table below shows the properties that a |
| node of this type may have. |
| |
| +------------+----------+-------------+---------------------------------------------------------------+ |
| | Property | Type | Required? | Description | |
| +============+==========+=============+===============================================================+ |
| | feed\_id | string | yes | Feed identifier assigned by DMaaP when the feed was created | |
| +------------+----------+-------------+---------------------------------------------------------------+ |
| |
| - ``ccsdk.nodes.ExternalTargetFeed``: This type represents a feed |
| created in an external DMaaP environment (i.e., an environment that |
| the plugin cannot access to make provisioning requests, such as a |
| shared corporate system). Nodes of this type are placed in a |
| blueprint so that other feed nodes of type ``ccsdk.nodes.Feed`` or |
| ``ccsdk.nodes.ExistingFeed`` can be set up to "bridge" to external |
| feeds by publishing data to the external feeds. The table below shows |
| the properties that a node of this type may have. |
| |
| +------------+----------+-------------+----------------------------------------------------------------+ |
| | Property | Type | Required? | Description | |
| +============+==========+=============+================================================================+ |
| | url | string | yes | The publish URL of the external feed. | |
| +------------+----------+-------------+----------------------------------------------------------------+ |
| | username | string | yes | The username to be used when delivering to the external feed | |
| +------------+----------+-------------+----------------------------------------------------------------+ |
| | userpw | string | yes | The password to be used when delivering to the external feed | |
| +------------+----------+-------------+----------------------------------------------------------------+ |
| |
| *Note: These properties are usually obtained by manually creating a feed |
| in the external DMaaP DR system and then creating a publisher for that |
| feed.* |
| |
| - ``ccsdk.nodes.ExternalSourceFeed``: This type represents a feed |
| created in an external DMaaP environment (i.e., an environment that |
| the plugin cannot access to makes provisioning requests, such as a |
| shared corporate system). Nodes of this type are place in a blueprint |
| so that they can be set up to "bridge" to other feed nodes of type |
| ``ccsdk.nodes.Feed`` or ``ccsdk.nodes.ExistingFeed``. This type has |
| no node properties, but when a bridge is set up, the url, username, |
| and password are attached to the node as runtime\_properties, using |
| the name of the target feed node as the top-level key. |
| |
| There are five relationship types for DMaaP Data Router: |
| |
| - ``ccsdk.relationships.publish_files``, used to indicate that the |
| relationship's source node sends is a publisher to the Data Router |
| feed represented by the relationship's target node. |
| - ``ccsdk.relationships.subscribe_to_files``, used to indicate that the |
| relationship's source node is a subscriber to the Data Router feed |
| represented by the relationship's target node. |
| - ``ccsdk.relationships.bridges_to``, used to indicate that the |
| relationship's source node (a ``ccsdk.nodes.Feed`` or |
| ``ccsdk.nodes.ExistingFeed``) should be set up to forward data |
| ("bridge") to the relationship's target feed (another |
| ``ccsdk.nodes.Feed`` or ``ccsdk.nodes.ExistingFeed``). |
| - ``ccsdk.relationships.bridges_to_external``, used to indicate that |
| the relationship's source node (a ``ccsdk.nodes.Feed`` or |
| ``ccsdk.nodes.ExistingFeed``) should be set up to forward data |
| ("bridge") to the relationship's target node (a feed in an external |
| DMaaP system, represented by a ``ccsdk.nodes.ExternalTargetFeed`` |
| node). |
| - ``ccsdk.relationships.bridges_from_external_to_internal``, used to |
| indicate the the relationship's source node (a feed in an external |
| DMaaP system, represented by a ``ccsdk.nodes.ExternalSourceFeed`` |
| node) should be set up to forward date ("bridge") to the |
| relationship's target node (an internal ONAP feed, represented by a |
| ``ccsdk.nodes.Feed`` or ``ccsdk.nodes.ExistingFeed`` node). |
| |
| The plugin code implements the lifecycle operations needed to create and |
| delete feeds and to add and remove publishers and subscribers. It also |
| implements the operations needed to set up bridging between feeds. |
| |
| Interaction with Other Plugins |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| |
| When creating a new feed or processing a reference to an existing feed, |
| the plugin operates independently of other plugins. |
| |
| When processing a ``ccsdk.relationships.publish_files`` relationship or |
| a ``ccsdk.relationships.subscribe_to_files`` relationship, this plugin |
| needs to obtain data from the source node and, in the case of |
| ``publish_files``, provide data to the source node. Certain conventions |
| are therefore needed for passing data between this plugin and the |
| plugins responsible for the source nodes in these relationships. In |
| Cloudify, the mechanism for sharing data among plugins is the |
| ``ctx.instance.runtime_properties`` dictionary associated with each |
| node. |
| |
| A given source node may have relationships with several feeds. For |
| example, an ONAP DCAE data collector might publish two different types |
| of data to two different feeds. An ONAP DCAE analytics module might |
| subscribe to one feed to get input for its processing and publish its |
| results to a different feed. When this DMaaP plugin and the plugin for |
| the source node exchange information, they need to do in a way that lets |
| them distinguish among different feeds. We do this through a simple |
| convention: for each source node to feed relationship, the source node |
| plugin will create a property in the source node's |
| ``runtime_properties`` dictionary. The name of the property will be the |
| same as the name of the target node of the relationship. For instance, |
| if a node has a ``publishes_files`` relationship with a target node |
| named ``feed00``, then the plugin that's responsible for managing the |
| source node with create an entry in the source node's |
| ``runtime_properties`` dictionary named ``feed00``. This entry itself |
| will be a dictionary. |
| |
| The content of this data exchange dictionary depends on whether the |
| source node is a publisher (i.e., the relationship is ``publish_files``) |
| or a subscriber (i.e., the relationship is ``subscribe_to_files``). |
| |
| For the ``publish_files`` relationship, the data exchange dictionary has |
| the following properties: |
| |
| +----------------+----------------------+--------------------------------------------------------------------------------------------------+ |
| | Property | Set by | Description | |
| +================+======================+==================================================================================================+ |
| | location | source node plugin | the DMaaP location for the publisher, used to set up routing | |
| +----------------+----------------------+--------------------------------------------------------------------------------------------------+ |
| | publish\_url | DMaaP plugin | the URL to which the publisher makes Data Router publish requests | |
| +----------------+----------------------+--------------------------------------------------------------------------------------------------+ |
| | log\_url | DMaaP plugin | the URL from which log data for the feed can be obtained | |
| +----------------+----------------------+--------------------------------------------------------------------------------------------------+ |
| | username | DMaaP plugin | the username (generated by the DMaaP plugin) the publisher uses to authenticate to Data Router | |
| +----------------+----------------------+--------------------------------------------------------------------------------------------------+ |
| | password | DMaaP plugin | the password (generated by the DMaaP plugin) the publisher uses to authenticate to Data Router | |
| +----------------+----------------------+--------------------------------------------------------------------------------------------------+ |
| |
| For the ``subscribe_to_files`` relationship, the data exchange |
| dictionary has the following properties: |
| |
| +-----------------+----------------------+-----------------------------------------------------------------------------------------+ |
| | Property | Set by | Description | |
| +=================+======================+=========================================================================================+ |
| | location | source node plugin | the DMaaP location for the subscriber, used to set up routing | |
| +-----------------+----------------------+-----------------------------------------------------------------------------------------+ |
| | delivery\_url | source node plugin | the URL to which the Data Router should deliver files | |
| +-----------------+----------------------+-----------------------------------------------------------------------------------------+ |
| | username | source node plugin | the username Data Router uses to authenticate to the subscriber when delivering files | |
| +-----------------+----------------------+-----------------------------------------------------------------------------------------+ |
| | password | source node plugin | the username Data Router uses to authenticate to the subscriber when delivering file | |
| +-----------------+----------------------+-----------------------------------------------------------------------------------------+ |
| |
| Plugin Support for DMaaP Message Router |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| |
| Plugin Types for DMaaP Message Router |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| |
| The Cloudify type definitions for DMaaP Message Router nodes and |
| relationships are defined in ```dmaap.yaml`` <./dmaap.yaml>`__. |
| |
| There are two node types for DMaaP Message Router: |
| |
| - ``ccsdk.nodes.Topic``: This type represents a topic that does not yet |
| exist and that should be created when the install workflow is run |
| against a blueprint that contains a node of this type. |
| |
| +----------------------+-----------+-------------+-----------------------------------------------------------------------------+ |
| | Property | Type | Required? | Description | |
| +======================+===========+=============+=============================================================================+ |
| | topic\_name | string | no | a name that uniquely identifies the feed (plugin will generate if absent) | |
| +----------------------+-----------+-------------+-----------------------------------------------------------------------------+ |
| | topic\_description | string | no | human-readable description of the feed | |
| +----------------------+-----------+-------------+-----------------------------------------------------------------------------+ |
| | txenable | boolean | no | flag indicating whether transactions are enabled for this topic | |
| +----------------------+-----------+-------------+-----------------------------------------------------------------------------+ |
| | replication\_case | string | no | type of replication required for the topic (defaults to no replication) | |
| +----------------------+-----------+-------------+-----------------------------------------------------------------------------+ |
| | global\_mr\_url | string | no | Global MR host name for replication to a global MR instance | |
| +----------------------+-----------+-------------+-----------------------------------------------------------------------------+ |
| |
| Note: In order to set up topics, a user should be familiar with message |
| router and how it is configured, and this README is not the place to |
| explain the details of message router. Here are a couple of pieces of |
| information that might be helpful. Currently, the allowed values for |
| ``replication_case`` are: |
| |
| - ``REPLICATION_NONE`` |
| - ``REPLICATION_EDGE_TO_CENTRAL`` |
| - ``REPLICATION_EDGE_TO_CENTRAL_TO_GLOBAL`` |
| - ``REPLICATION_CENTRAL_TO_EDGE`` |
| - ``REPLICATION_CENTRAL_TO_GLOBAL`` |
| - ``REPLICATION_GLOBAL_TO_CENTRAL`` |
| - ``REPLICATION_GLOBAL_TO_CENTRAL_TO_EDGE`` |
| |
| The ``global_mr_url`` is actually a host name, not a full URL. It points |
| to a host in a global message router cluster. (A 'global' message router |
| cluster is one that's not part of ONAP.) |
| |
| - ``ccsdk.nodes.ExistingTopic``: This type represents a topic that |
| already exists. Nodes of this type are placed in a blueprint so that |
| other nodes in the blueprint can be set up as publishers or |
| subscribers to the topic. The table below shows the properties that a |
| node of this type may have. |
| |
| +------------+----------+-------------+--------------------------------------------+ |
| | Property | Type | Required? | Description | |
| +============+==========+=============+============================================+ |
| | fqtn | string | yes | fully-qualified topic name for the topic | |
| +------------+----------+-------------+--------------------------------------------+ |
| |
| Interaction with Other Plugins |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| |
| When creating a new topic or processing a reference to an existing |
| topic, the plugin operates independently of other plugins. |
| |
| When processing a ``ccsdk.relationships.publish_events`` relationship or |
| a ``ccsdk.relationships.subscribe_to_events`` relationship, this plugin |
| needs to obtain data from and provide data to the source node. Certain |
| conventions are therefore needed for passing data between this plugin |
| and the plugins responsible for the source nodes in these relationships. |
| In Cloudify, the mechanism for sharing data among plugins is the |
| ``ctx.instance.runtime_properties`` dictionary associated with each |
| node. |
| |
| A given source node may have relationships with several topics. For |
| example, an ONAP DCAE analytics module might subscribe to one topic to |
| get input for its processing and publish its results to a different |
| topic. When this DMaaP plugin and the plugin for the source node |
| exchange information, they need to do in a way that lets them |
| distinguish among different feeds. We do this through a simple |
| convention: for each source node to topic relationship, the source node |
| plugin will create a property in the source node's |
| ``runtime_properties`` dictionary. The name of the property will be the |
| same as the name of the target node of the relationship. For instance, |
| if a node has a ``publishes_events`` relationship with a target node |
| named ``topic00``, then the plugin that's responsible for managing the |
| source node with create an entry in the source node's |
| ``runtime_properties`` dictionary named ``topic00``. This entry itself |
| will be a dictionary. |
| |
| For both types of relationship, the data exchange dictionary has the |
| following properties: |
| |
| +----------------+----------------------+----------------------------------------------------------------------------------+ |
| | Property | Set by | Description | |
| +================+======================+==================================================================================+ |
| | location | source node plugin | the DMaaP location for the publisher or subscriber, used to set up routing | |
| +----------------+----------------------+----------------------------------------------------------------------------------+ |
| | client\_role | source node plugin | the AAF client role that's requesting publish or subscribe access to the topic | |
| +----------------+----------------------+----------------------------------------------------------------------------------+ |
| | topic\_url | DMaaP plugin | the URL for accessing the topic to publish or receive events | |
| +----------------+----------------------+----------------------------------------------------------------------------------+ |
| |
| Interaction with Consul configuration store |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| |
| In addition to storing the results of DMaaP Data Router and DMaaP |
| Message Router provisioning operations in ``runtime_properties``, the |
| DMaaP plugin also stores these results into the ONAP configuration |
| store, which resides in a `Consul key-value |
| store <https://www.consul.io/>`__. This allows DMaaP clients (components |
| that act as publishers, subscribers, or both) to retrieve their DMaaP |
| configuration information from Consul, rather than having the plugin |
| that deploys the client directly configure the client using data in |
| ``runtime_properties``. |
| |
| The ``runtime_properties`` for a client must contain a property called |
| ``service_component_name``. If this property is not present, the plugin |
| will raise a NonRecoverableError and cause the installation to fail. |
| |
| If ``service_component_name`` is present, then the plugin will use a |
| Consul key consisting of the value of ``service_component_name`` |
| prepended to the fixed string ``:dmaap``. For example, if the |
| ``service_component_name`` is ``client123``, the plugin will use |
| ``client123:dmaap`` as the key for storing DMaaP information into |
| Consul. Information for all of the feeds and topics for a client are |
| stored under the same key. |
| |
| The value stored is a nested JSON object. At the top level of the object |
| are properties representing each topic or feed for which the component |
| is a publisher or subscriber. The name of the property is the node name |
| of the target feed or topic. The value of the property is another JSON |
| object that corresponds to the dictionary that the plugin created in |
| ``runtime_properties`` corresponding to the target feed or topic. Note |
| that the information in Consul includes all of the properties for the |
| feed or topic, those set by the source node plugin as well as those set |
| by the DMaaP plugin. |
| |
| Examples: |
| |
| Data Router publisher, target feed ``feed00``: |
| |
| :: |
| |
| { |
| "feed00": { |
| "username": "rC9QR51I", |
| "log_url": "https://dmaap.example.com/feedlog/972", |
| "publish_url": "https://dmaap.example.com/publish/972", |
| "location": "loc00", |
| "password": "QOQeUh5KLR", |
| "publisher_id": "972.360gm" |
| } |
| } |
| |
| Data Router subscriber, target feed ``feed01``: |
| |
| :: |
| |
| { |
| "feed01": { |
| "username": "drdeliver", |
| "password": "1loveDataR0uter", |
| "location": "loc00", |
| "delivery_url": "https://example.com/whatever", |
| "subscriber_id": "1550" |
| } |
| } |
| |
| Message Router publisher to ``topic00``, subscriber to ``topic01``. Note |
| how each topic appears as a top-level property in the object. |
| |
| :: |
| |
| { |
| "topic00": { |
| "topic_url": "https://dmaap.example.com:3905/events/org.onap.ccsdk.dmaap.FTL2.outboundx", |
| "client_role": "org.onap.ccsdk.member", |
| "location": "loc00", |
| "client_id": "1494621774522" |
| }, |
| "topic01": { |
| "topic_url": "https://dmaap.example.com:3905/events/org.onap.ccsdk.dmaap.FTL2.inboundx", |
| "client_role": "org.onap.ccsdk.member", |
| "location": "loc00", |
| "client_id": "1494621778627" |
| } |
| } |
| |
| Packaging and installing |
| ~~~~~~~~~~~~~~~~~~~~~~~~ |
| |
| The DMaaP plugin is meant to be used as a `Cloudify managed |
| plugin <http://docs.getcloudify.org/3.4.0/plugins/using-plugins/>`__. |
| Managed plugins are packaged using |
| ```wagon`` <https://github.com/cloudify-cosmo/wagon>`__. |
| |
| To package this plugin, executing the following command in the top-level |
| directory of this plugin, from a Python environment in which ``wagon`` |
| has been installed: |
| |
| :: |
| |
| wagon create -s . -r -o /path/to/directory/for/wagon/output |
| |
| Once the wagon file is built, it can be uploaded to a Cloudify Manager |
| host using the ``cfy plugins upload`` command described in the |
| documentation above. |
| |
| Managed plugins can also be loaded at the time a Cloudify Manager host |
| is installed, via the installation blueprint and inputs file. We expect |
| that this plugin will be loaded at Cloudify Manager installation time, |
| and that ``cfy plugins upload`` will be used only for delivering patches |
| between releases. |
| |
| Configuration |
| ~~~~~~~~~~~~~ |
| |
| The plugin needs to be configured with certain parameters needed to |
| access the DMaaP Bus Controller. In keeping with the ONAP architecture, |
| this information is stored in Consul. |
| |
| The plugin finds the address and port of the DMaaP Bus Controller using |
| the Consul service discovery facility. The plugin expects the Bus |
| Controller to be registered under the name ``dmaap_bus_controller``. |
| |
| Additional parameters come from the ``dmaap`` key in the Cloudify |
| Manager's Consul configuration, which is stored in the Consul KV store |
| under the key name 'cloudify\_manager'. The table below lists the |
| properties in the configuration: |
| |
| +----------------+----------+-------------+--------------+---------------------------------------------------------------------------------------------+ |
| | Property | Type | Required? | Default | Description | |
| +================+==========+=============+==============+=============================================================================================+ |
| | ``username`` | string | Yes | (none) | The username for logging into DMaaP Bus Controller | |
| +----------------+----------+-------------+--------------+---------------------------------------------------------------------------------------------+ |
| | ``password`` | string | Yes | (none) | The password for logging into DMaaP Bus Controller | |
| +----------------+----------+-------------+--------------+---------------------------------------------------------------------------------------------+ |
| | ``owner`` | string | Yes | (none) | The name to be used as the owner for entities created by the plugin | |
| +----------------+----------+-------------+--------------+---------------------------------------------------------------------------------------------+ |
| | ``protocol`` | string | No | ``https`` | The protocol (URL scheme) used to access the DMaaP bus controller (``http`` or ``https``) | |
| +----------------+----------+-------------+--------------+---------------------------------------------------------------------------------------------+ |
| | ``path`` | string | No | ``webapi`` | The path to the root of the DMaaP Bus Controller API endpoint | |
| +----------------+----------+-------------+--------------+---------------------------------------------------------------------------------------------+ |
| |
| Here is an example of a Cloudify Manager configuration object showing |
| only the ``dmaap`` key: |
| |
| :: |
| |
| { |
| "dmaap": { |
| "username": "dmaap.client@ccsdkorch.onap.org", |
| "password": "guessmeifyoucan" |
| "owner": "ccsdkorc" |
| }, |
| |
| (other configuration here) |
| |
| } |
| |