Ralph Knag | 1fca6ac | 2017-12-05 12:05:57 -0500 | [diff] [blame] | 1 | .. This work is licensed under a Creative Commons Attribution 4.0 International License.
|
| 2 | .. http://creativecommons.org/licenses/by/4.0
|
| 3 |
|
| 4 | .. _cdap-specification:
|
| 5 |
|
| 6 | Component specification (CDAP)
|
| 7 | ==============================
|
| 8 |
|
| 9 | The CDAP component specification contains the following groups of
|
| 10 | information. Many of these are common to both CDAP and Docker components
|
| 11 | and are therefore described in the common specification.
|
| 12 |
|
| 13 | - :any:`Metadata <metadata>`
|
| 14 | - :any:`Interfaces <interfaces>` including the
|
| 15 | associated :any:`Data Formats <data-formats>`
|
| 16 | - :any:`Parameters <parameters>` - for specifying parameters in your
|
| 17 | AppConfig, AppPreferences, and ProgramPreferences to the Designer and
|
| 18 | Policy. This of course is CDAP-specific and is described below.
|
| 19 | - :any:`Auxiliary Details <auxiliary-details>`
|
| 20 | - :any:`List of artifacts <artifacts>`
|
| 21 |
|
| 22 | Current Limitations and TODOs
|
| 23 | -----------------------------
|
| 24 |
|
| 25 | - The integration of DMD is likely to significantly change the
|
Ralph Knag | d2cd31b | 2018-04-02 16:27:46 -0400 | [diff] [blame] | 26 | :any:`Interfaces <interfaces>` section in this specification.
|
Ralph Knag | 1fca6ac | 2017-12-05 12:05:57 -0500 | [diff] [blame] | 27 |
|
| 28 | .. _parameters:
|
| 29 |
|
| 30 | Parameters
|
| 31 | ----------
|
| 32 |
|
| 33 | There is a ``parameters`` section in your component specification. This
|
| 34 | section contains three optional keys: `app_config <#appconfig>`__,
|
| 35 | `app_preferences <#apppreferences>`__, and
|
Ralph Knag | d2cd31b | 2018-04-02 16:27:46 -0400 | [diff] [blame] | 36 | `program_preferences <#programpreferences>`__:
|
Ralph Knag | 1fca6ac | 2017-12-05 12:05:57 -0500 | [diff] [blame] | 37 |
|
| 38 | ::
|
| 39 |
|
| 40 | "parameters" : {
|
| 41 | "app_config" : [ ...],
|
| 42 | "app_preferences" : [ ...],
|
| 43 | "program_preferences" : [...]
|
| 44 | // any additional keys are ignored
|
| 45 | }
|
| 46 |
|
| 47 | - Each section details the parameters that are a part of each of these
|
| 48 | CDAP constructs (see below).
|
| 49 | - All such parameters will be exposed to the designer and to policy for
|
| 50 | override.
|
| 51 | - These parameters should have default values specified by the
|
| 52 | component developer where necessary, i.e., parameters that *must*
|
| 53 | come from the designer/policy should not have defaults.
|
| 54 | - All of these keys are optional because not every CDAP application
|
| 55 | uses preferences and not every application uses the AppConfig.
|
| 56 | However, you should specify at least one, or else your application
|
| 57 | will have no parameters exposed to policy or to the DCAE designer,
|
| 58 | which means it would be non-configurable.
|
| 59 | - Despite the AppConfig being optional to *specify* in the case that
|
| 60 | you have no parameters in your AppConfig, it is *required for
|
| 61 | processing* in your application. That is because the DCAE platform
|
| 62 | will place important information into your AppConfig as discussed
|
| 63 | below.
|
| 64 |
|
| 65 | Parameter
|
| 66 | ~~~~~~~~~
|
| 67 |
|
| 68 | The following CDAP specific definitions use ``param1`` to refer to the
|
| 69 | common parameter layout in
|
| 70 | :any:`Parameter <parameters>`
|
| 71 |
|
| 72 | AppConfig
|
| 73 | ~~~~~~~~~
|
| 74 |
|
| 75 | The ``app_config`` key refers to the `CDAP AppConfig <http://docs.cask.co/cdap/current/en/reference-manual/http-restful-api/configuration.html>`_.
|
| 76 | It is expected to be a JSON:
|
| 77 |
|
| 78 | ::
|
| 79 |
|
| 80 | "app_config" : [ // list of JSON
|
| 81 | param1, // common parameter layout
|
| 82 | ...
|
| 83 | ]
|
| 84 |
|
| 85 | Unfortunately, at the time of writing, the AppConfig is a Java map of
|
| 86 | ``string:string``, which means you cannot have more complex structures
|
| 87 | (than string) as any value in your AppConfig. However, there is a way to
|
| 88 | bypass this constraint: you can pass a JSON by encoding the JSON as a
|
| 89 | string. E.g., the ``json.dumps()`` and it’s converse ``loads`` methods
|
| 90 | in Python:
|
| 91 |
|
| 92 | ::
|
| 93 |
|
| 94 | >>> import json
|
| 95 | >>> json.dumps({"foo" : "bar"}) # This is a real JSON
|
| 96 | '{"foo": "bar"}' # It is now a string: pass this in as your parameter value
|
| 97 | >>> json.loads('{"foo": "bar"}') # Do the equivelent of this in your application
|
| 98 | {u'foo': u'bar'} # ...and you'll get back a real JSON
|
| 99 | >>>
|
| 100 |
|
| 101 | The final AppConfig (after the designer and policy override parameter
|
| 102 | values) is passed into CDAP’s AppConfig API when starting the
|
| 103 | application.
|
| 104 |
|
| 105 |
|
| 106 | AppPreferences
|
| 107 | ~~~~~~~~~~~~~~
|
| 108 |
|
| 109 | In addition to the CDAP AppConfig, the platform supports `Application Preferences <http://docs.cask.co/cdap/current/en/reference-manual/http-restful-api/preferences.html#set-preferences>`_.
|
| 110 | The format of the ``app_preferences`` value is the same as the above:
|
| 111 |
|
| 112 | ::
|
| 113 |
|
| 114 | "app_preferences" : [ // list of JSON
|
| 115 | param1, // common parameter layout
|
| 116 | ...
|
| 117 | ]
|
| 118 |
|
| 119 | The final Application Preferences JSON (after the designer and policy
|
| 120 | override parameter values) is passed into CDAP’s Preferences API when
|
| 121 | starting your application.
|
| 122 |
|
| 123 |
|
| 124 | ProgramPreferences
|
| 125 | ~~~~~~~~~~~~~~~~~~
|
| 126 |
|
| 127 | Preferences can also be specified `per program <http://docs.cask.co/cdap/current/en/reference-manual/http-restful-api/lifecycle.html#program-lifecycle>`_
|
| 128 | in CDAP. This key’s value is a list of JSON with the following format:
|
| 129 |
|
| 130 | ::
|
| 131 |
|
| 132 | "program_preferences" : [ // note: this is a list of JSON
|
| 133 | {
|
| 134 | "program_id" : "program name 1", // the name of this CDAP program
|
| 135 | "program_type" : "e.g., flows", // "must be one of flows, mapreduce, schedules, spark, workflows, workers, or services",
|
| 136 | "program_pref" : [ // list of JSON
|
| 137 | param1, // common parameter layout
|
| 138 | ...
|
| 139 | ]
|
| 140 | },
|
| 141 | // repeat for each program that receives a program_preferences JSON
|
| 142 | ]
|
| 143 |
|
| 144 | Each ``program_pref`` JSON is passed into the CDAP API as the preference
|
| 145 | for ``program_id``.
|
| 146 |
|
| 147 | NOTE: for CDAP, this section is very likely to change when DMD is
|
Ralph Knag | d2cd31b | 2018-04-02 16:27:46 -0400 | [diff] [blame] | 148 | available. The *future* vision is
|
Ralph Knag | 1fca6ac | 2017-12-05 12:05:57 -0500 | [diff] [blame] | 149 | that you would publish your data as a series of files on HDFS, and DMD
|
| 150 | will pick them up and send them to the appropriate DMaaP feeds or
|
| 151 | directly when needed.
|
| 152 |
|
| 153 | .. _auxiliary-details:
|
| 154 |
|
| 155 | Auxiliary Details
|
| 156 | -----------------
|
| 157 |
|
| 158 | *auxiliary* contains details about CDAP specific parameters.
|
| 159 |
|
| 160 | +----------------------+----------------------+----------------------+
|
| 161 | | Property Name | Type | Description |
|
| 162 | +======================+======================+======================+
|
| 163 | | streamname | string | *Required*. |
|
| 164 | +----------------------+----------------------+----------------------+
|
| 165 | | artifact_name | string | |
|
| 166 | +----------------------+----------------------+----------------------+
|
| 167 | | artifact_version | string | the version of your |
|
| 168 | | | | CDAP JAR artifact |
|
| 169 | +----------------------+----------------------+----------------------+
|
| 170 | | namespace | string | the CDAP namespace |
|
| 171 | | | | to deploy into, |
|
| 172 | | | | default is ‘default’ |
|
| 173 | +----------------------+----------------------+----------------------+
|
| 174 | | programs | array | contains each CDAP |
|
| 175 | | | | entity represented |
|
| 176 | | | | in the artifact |
|
| 177 | +----------------------+----------------------+----------------------+
|
| 178 | | program_type | string | CDAP entity (eg |
|
| 179 | | | | “flows”) |
|
| 180 | +----------------------+----------------------+----------------------+
|
| 181 | | program_id | string | name of CDAP entity |
|
| 182 | | | | (eg “WhoFlow”) |
|
| 183 | +----------------------+----------------------+----------------------+
|
| 184 |
|
| 185 | Example:
|
| 186 |
|
| 187 | .. code:: json
|
| 188 |
|
| 189 | "auxiliary": {
|
| 190 | "streamname" : "who",
|
| 191 | "artifact_name" : "HelloWorld",
|
| 192 | "artifact_version" : "3.4.3",
|
| 193 | "namespace" : "hw",
|
| 194 | "programs" : [
|
| 195 | {"program_type" : "flows", "program_id" : "WhoFlow"},
|
| 196 | {"program_type" : "services", "program_id" : "Greeting"},
|
| 197 | ...
|
| 198 | ],
|
| 199 | }
|
| 200 |
|
| 201 | The ``programs`` key is identical to the ``program_preferences`` key
|
| 202 | discussed `above <#programpreferences>`__ except:
|
| 203 |
|
| 204 | - each JSON in the list does not contain ``program_pref``
|
| 205 | - this is required! You must include all of your programs in this, as
|
| 206 | it is used to start each program as well as for DCAE to perform
|
| 207 | periodic healthchecks on your application. Don’t forget about your
|
| 208 | services; they are programs too.
|