Added support for Consul/CBS and multiple DFCs
Issue-ID: INT-1155
Change-Id: I3c1ed2f6072655c4396e406ddfd490d3786fe4d6
Signed-off-by: BjornMagnussonXA <bjorn.magnusson@est.tech>
diff --git a/test/mocks/datafilecollector-testharness/simulator-group/.gitignore b/test/mocks/datafilecollector-testharness/simulator-group/.gitignore
index 48e8e80..fada972 100644
--- a/test/mocks/datafilecollector-testharness/simulator-group/.gitignore
+++ b/test/mocks/datafilecollector-testharness/simulator-group/.gitignore
@@ -1,6 +1,5 @@
-configuration
-tls
docker-compose.yml
node_modules
package.json
-prepare-images.sh
\ No newline at end of file
+package-lock.json
+.tmp*
\ No newline at end of file
diff --git a/test/mocks/datafilecollector-testharness/simulator-group/Dockerfile-sim-monitor b/test/mocks/datafilecollector-testharness/simulator-group/Dockerfile-sim-monitor
deleted file mode 100644
index 145d2d9..0000000
--- a/test/mocks/datafilecollector-testharness/simulator-group/Dockerfile-sim-monitor
+++ /dev/null
@@ -1,15 +0,0 @@
-#Image for monitor simulator
-
-FROM node:8
-
-WORKDIR /app
-
-COPY sim-monitor.js ./
-COPY package*.json ./
-
-RUN npm install express
-RUN npm install argparse
-
-EXPOSE 9999
-
-CMD node /app/sim-monitor.js
\ No newline at end of file
diff --git a/test/mocks/datafilecollector-testharness/simulator-group/README.md b/test/mocks/datafilecollector-testharness/simulator-group/README.md
index dc8f286..5a51d8a 100644
--- a/test/mocks/datafilecollector-testharness/simulator-group/README.md
+++ b/test/mocks/datafilecollector-testharness/simulator-group/README.md
@@ -1,9 +1,69 @@
-#Introduction
+###Introduction
The purpose of the "simulator-group" is to run all containers in one go with specified behavior.
-Mainly this is needed for CSIT tests but can be used also for local testing.
+Mainly this is needed for CSIT tests and for auto test but can be used also for manual testing of dfc both as an java-app
+or as a manually started container. Instead of running the simulators manually as described below the auto-test cases
+can be executed together with a java-app or a manaully started container.
+
+In general these steps are needed to run the simulator group and dfc
+
+1. Build the simulator images
+2. Edit simulator env variables (to adapt the behavior of simulators)
+3. Configure consul
+4. Start the simulator monitor (to view the simulator stats)
+5. Start the simulators
+6. Start dfc
+
+###Overview of the simulators.
+There are 5 different types of simulators. For futher details, see the README.md in each simulator dir.
+
+1. The MR simulator emits fileready events, upon poll requests, with new and historice file references.
+It is possible to configire the change identifier and file prefixes for these identifiers and for which consumer groups
+these change identifier shall be generated. It is also possible to configure the number of events and files to generate and
+from which ftp servers the files shall be fetched from.
+2. The DR simulator handles the publish queries (to check if a file has previously been published) and the
+actual publish request (which results in a redirect to the DR REDIR simulator. It keeps a 'db' of published files updated by the DR REDIR simulator.
+It is possible to configure 1 or more feeds along with the accepted filename prefixes for each feed. It is also possible
+to configure the responses for the publish queries and publish requests.
+3. The DR REDIR simulator handles the redirect request for publish from the DR simulator. All accepted files will be stored as and empty
+file with a file name concatenated from the published file name + file size + feed id.
+It is possible to configure 1 or more feeds along with the accepted filename prefixes for each feed. It is also possible
+to configure the responses for the publish requests.
+4. The SFTP simulator(s) handles the ftp download requests. 5 of these simulators are always started and in the MR sim it is
+possible to configure the distrubution of files over these 5 servers (from 1 up to 5 severs). At start of the server, the server is
+populated with files to download.
+5. The FTPS simulator(s) is the same as the SFTP except that it using the FTPS protocol.
-###Preparation
+### Build the simulator images
+Run the script `prepare-images.sh` to build the docker images for MR, DR and FTPS servers.
+
+###Edit simulator env variables
+
+
+
+
+###Summary of scripts and files
+`consul_config.sh` - Convert a json config file to work with dfc when manually started as java-app or container and then add that json to Consul.
+
+`docker-compose-setup.sh` - Sets environment variables for the simulators and start the simulators with that settings.
+
+`docker-compose-template.yml` - A docker compose template with environment variables setting. Used for producing a docker-compose file to defined the simulator containers.
+
+`prepare-images.sh` - Script to build all needed simulator images.
+
+`setup-ftp-files-for-image.sh` - Script executed in the ftp server to create files for download.
+
+`sim-monitor-start.sh` - Script to install needed packages and start the simulator monitor.
+
+`sim-monitor.js` - The source file the simulator monitor.
+
+`simulators-kill.sh` - Script to kill all the simulators
+
+`simulators-start.sh` - Script to start all the simulators. All env variables need to be set prior to executing the script.
+
+
+
+###Preparation
Do the manual steps to prepare the simulator images
Build the mr-sim image.
@@ -33,8 +93,8 @@
Edit the `docker-compose-setup.sh` (or create a copy) to setup the env variables to the desired test behavior for each simulators.
See each simulator to find a description of the available settings (DR_TC, DR_REDIR_TC and MR_TC).
The following env variables shall be set (example values).
-Note that NUM_FTPFILES and NUM_PNFS controls the number of ftp files created in the ftp servers.
-A total of NUM_FTPFILES * NUM_PNFS ftp files will be created in each ftp server (4 files in the below example).
+Note that NUM_FTPFILES and NUM_PNFS controls the number of ftp files created in the ftp servers.
+A total of NUM_FTPFILES * NUM_PNFS ftp files will be created in each ftp server (4 files in the below example).
Large settings will be time consuming at start of the servers.
Note that the number of files must match the number of file references emitted from the MR sim.
@@ -42,7 +102,7 @@
DR_REDIR_TC="--tc normal" #Normal behavior of the DR redirect sim
-MR_TC="--tc100" #One 1 MB file in one event, once.
+MR_TC="--tc100" #One 1 MB file in one event, once.
BC_TC="" #Not in use yet
@@ -52,7 +112,7 @@
To minimize the number of ftp file creation, the following two variables can be configured in the same file.
FILE_SIZE="1MB" #File size for FTP file (1KB, 1MB, 5MB, 50MB or ALL)
-FTP_TYPE="SFTP" #Type of FTP files to generate (SFTP, FTPS or ALL)
+FTP_TYPE="SFTP" #Type of FTP files to generate (SFTP, FTPS or ALL)
If `FTP_TYPE` is set to `ALL`, both ftp servers will be populated with the same files. If set to `SFTP` or `FTPS` then only the server serving that protocol will be populated with files.
@@ -73,19 +133,11 @@
Start DFC by the following cmd: `docker run -d --network="host" --name dfc_app <dfc-image> `
`<dfc-image>` could be either the locally built image `onap/org.onap.dcaegen2.collectors.datafile.datafile-app-server`
-or the one in nexus `nexus3.onap.org:10001/onap/org.onap.dcaegen2.collectors.datafile.datafile-app-server`.
+or the one in nexus `nexus3.onap.org:10001/onap/org.onap.dcaegen2.collectors.datafile.datafile-app-server`.
-###Simulator monitor
-Start the simulator monitor server with `sim-monitor-start.sh` and the open a browser with the url `localhost:9999/mon`
-to see the statisics page with data from MR sim, DR sim and DR redir sim.
-Or run as a container, build image first. Note, does not work on Mac.
-
-`cp ../dr-sim/package.json .`
-
-`docker build -t sim-mon:latest -f Dockerfile-sim-monitor .`
-
-Then run it, `docker run --network="host" --name sim-mon -it -d sim-mon:latest`
-
-Stop it with `docker stop sim-mon` and if desired, remove the container by `docker rm sim-mon`
+###Start the simulator monitor
+Start the simulator monitor server with `node sim-monitor.js` on the cmd line and the open a browser with the url `localhost:9999/mon`
+to see the statisics page with data from DFC(ss), MR sim, DR sim and DR redir sim.
+If needed run 'npm install express' first
diff --git a/test/mocks/datafilecollector-testharness/simulator-group/consul/c12_feed1_PM.json b/test/mocks/datafilecollector-testharness/simulator-group/consul/c12_feed1_PM.json
new file mode 100644
index 0000000..282085c
--- /dev/null
+++ b/test/mocks/datafilecollector-testharness/simulator-group/consul/c12_feed1_PM.json
@@ -0,0 +1,23 @@
+{
+ "dmaap.ftpesConfig.keyCert":"config/dfc.jks",
+ "dmaap.ftpesConfig.keyPassword":"secret",
+ "dmaap.ftpesConfig.trustedCa":"config/ftp.jks",
+ "dmaap.ftpesConfig.trustedCaPassword":"secret",
+ "dmaap.security.trustStorePath":"change it",
+ "dmaap.security.trustStorePasswordPath":"trustStorePasswordPath",
+ "dmaap.security.keyStorePath":"keyStorePath",
+ "dmaap.security.keyStorePasswordPath":"change it",
+ "dmaap.security.enableDmaapCertAuth":"false",
+ "dmaap.dmaapProducerConfiguration" : {
+ "changeIdentifier":"PM_MEAS_FILES",
+ "feedName":"feed01"
+ },
+ "streams_subscribes":{
+ "dmaap_subscriber":{
+ "dmmap_info":{
+ "topic_url":"http://dradmin:dradmin@mrsim:2222/events/unauthenticated.VES_NOTIFICATION_OUTPUT/OpenDcae-c12/C12"
+ },
+ "type":"message_router"
+ }
+ }
+}
\ No newline at end of file
diff --git a/test/mocks/datafilecollector-testharness/simulator-group/consul/c12_feed1_PM_feed2_CTR.json b/test/mocks/datafilecollector-testharness/simulator-group/consul/c12_feed1_PM_feed2_CTR.json
new file mode 100644
index 0000000..2e4f62c
--- /dev/null
+++ b/test/mocks/datafilecollector-testharness/simulator-group/consul/c12_feed1_PM_feed2_CTR.json
@@ -0,0 +1,29 @@
+{
+ "dmaap.ftpesConfig.keyCert": "config/dfc.jks",
+ "dmaap.ftpesConfig.keyPassword": "secret",
+ "dmaap.ftpesConfig.trustedCa": "config/ftp.jks",
+ "dmaap.ftpesConfig.trustedCaPassword": "secret",
+ "dmaap.security.trustStorePath": "change it",
+ "dmaap.security.trustStorePasswordPath": "trustStorePasswordPath",
+ "dmaap.security.keyStorePath": "keyStorePath",
+ "dmaap.security.keyStorePasswordPath": "change it",
+ "dmaap.security.enableDmaapCertAuth": "false",
+ "dmaap.dmaapProducerConfiguration": [
+ {
+ "changeIdentifier": "PM_MEAS_FILES",
+ "feedName": "feed01"
+ },
+ {
+ "changeIdentifier": "CTR_FILES",
+ "feedName": "feed02"
+ }
+ ],
+ "streams_subscribes": {
+ "dmaap_subscriber": {
+ "dmmap_info": {
+ "topic_url": "http://dradmin:dradmin@mrsim:2222/events/unauthenticated.VES_NOTIFICATION_OUTPUT/OpenDcae-c12/C12"
+ },
+ "type": "message_router"
+ }
+ }
+}
\ No newline at end of file
diff --git a/test/mocks/datafilecollector-testharness/simulator-group/consul/c12_feed2_CTR.json b/test/mocks/datafilecollector-testharness/simulator-group/consul/c12_feed2_CTR.json
new file mode 100644
index 0000000..7eeed24
--- /dev/null
+++ b/test/mocks/datafilecollector-testharness/simulator-group/consul/c12_feed2_CTR.json
@@ -0,0 +1,23 @@
+{
+ "dmaap.ftpesConfig.keyCert":"config/dfc.jks",
+ "dmaap.ftpesConfig.keyPassword":"secret",
+ "dmaap.ftpesConfig.trustedCa":"config/ftp.jks",
+ "dmaap.ftpesConfig.trustedCaPassword":"secret",
+ "dmaap.security.trustStorePath":"change it",
+ "dmaap.security.trustStorePasswordPath":"trustStorePasswordPath",
+ "dmaap.security.keyStorePath":"keyStorePath",
+ "dmaap.security.keyStorePasswordPath":"change it",
+ "dmaap.security.enableDmaapCertAuth":"false",
+ "dmaap.dmaapProducerConfiguration" : {
+ "changeIdentifier":"CTR_MEAS_FILES",
+ "feedName":"feed02"
+ },
+ "streams_subscribes":{
+ "dmaap_subscriber":{
+ "dmmap_info":{
+ "topic_url":"http://dradmin:dradmin@mrsim:2222/events/unauthenticated.VES_NOTIFICATION_OUTPUT/OpenDcae-c12/C12"
+ },
+ "type":"message_router"
+ }
+ }
+}
\ No newline at end of file
diff --git a/test/mocks/datafilecollector-testharness/simulator-group/consul/c12_feed2_CTR_feed3_LOG_TEMP.json b/test/mocks/datafilecollector-testharness/simulator-group/consul/c12_feed2_CTR_feed3_LOG_TEMP.json
new file mode 100644
index 0000000..83be949
--- /dev/null
+++ b/test/mocks/datafilecollector-testharness/simulator-group/consul/c12_feed2_CTR_feed3_LOG_TEMP.json
@@ -0,0 +1,33 @@
+{
+ "dmaap.ftpesConfig.keyCert": "config/dfc.jks",
+ "dmaap.ftpesConfig.keyPassword": "secret",
+ "dmaap.ftpesConfig.trustedCa": "config/ftp.jks",
+ "dmaap.ftpesConfig.trustedCaPassword": "secret",
+ "dmaap.security.trustStorePath": "change it",
+ "dmaap.security.trustStorePasswordPath": "trustStorePasswordPath",
+ "dmaap.security.keyStorePath": "keyStorePath",
+ "dmaap.security.keyStorePasswordPath": "change it",
+ "dmaap.security.enableDmaapCertAuth": "false",
+ "dmaap.dmaapProducerConfiguration": [
+ {
+ "changeIdentifier": "CTR_MEAS_FILES",
+ "feedName": "feed02"
+ },
+ {
+ "changeIdentifier": "LOG_FILES",
+ "feedName": "feed03"
+ },
+ {
+ "changeIdentifier": "TEMP_FILES",
+ "feedName": "feed03"
+ }
+ ],
+ "streams_subscribes": {
+ "dmaap_subscriber": {
+ "dmmap_info": {
+ "topic_url": "http://dradmin:dradmin@mrsim:2222/events/unauthenticated.VES_NOTIFICATION_OUTPUT/OpenDcae-c12/C12"
+ },
+ "type": "message_router"
+ }
+ }
+}
\ No newline at end of file
diff --git a/test/mocks/datafilecollector-testharness/simulator-group/consul/c12_feed2_PM.json b/test/mocks/datafilecollector-testharness/simulator-group/consul/c12_feed2_PM.json
new file mode 100644
index 0000000..a366b4c
--- /dev/null
+++ b/test/mocks/datafilecollector-testharness/simulator-group/consul/c12_feed2_PM.json
@@ -0,0 +1,23 @@
+{
+ "dmaap.ftpesConfig.keyCert":"config/dfc.jks",
+ "dmaap.ftpesConfig.keyPassword":"secret",
+ "dmaap.ftpesConfig.trustedCa":"config/ftp.jks",
+ "dmaap.ftpesConfig.trustedCaPassword":"secret",
+ "dmaap.security.trustStorePath":"change it",
+ "dmaap.security.trustStorePasswordPath":"trustStorePasswordPath",
+ "dmaap.security.keyStorePath":"keyStorePath",
+ "dmaap.security.keyStorePasswordPath":"change it",
+ "dmaap.security.enableDmaapCertAuth":"false",
+ "dmaap.dmaapProducerConfiguration" : {
+ "changeIdentifier":"PM_MEAS_FILES",
+ "feedName":"feed02"
+ },
+ "streams_subscribes":{
+ "dmaap_subscriber":{
+ "dmmap_info":{
+ "topic_url":"http://dradmin:dradmin@mrsim:2222/events/unauthenticated.VES_NOTIFICATION_OUTPUT/OpenDcae-c12/C12"
+ },
+ "type":"message_router"
+ }
+ }
+}
diff --git a/test/mocks/datafilecollector-testharness/simulator-group/consul/c12_feed3_PM_CTR.json b/test/mocks/datafilecollector-testharness/simulator-group/consul/c12_feed3_PM_CTR.json
new file mode 100644
index 0000000..eca72be
--- /dev/null
+++ b/test/mocks/datafilecollector-testharness/simulator-group/consul/c12_feed3_PM_CTR.json
@@ -0,0 +1,29 @@
+{
+ "dmaap.ftpesConfig.keyCert": "config/dfc.jks",
+ "dmaap.ftpesConfig.keyPassword": "secret",
+ "dmaap.ftpesConfig.trustedCa": "config/ftp.jks",
+ "dmaap.ftpesConfig.trustedCaPassword": "secret",
+ "dmaap.security.trustStorePath": "change it",
+ "dmaap.security.trustStorePasswordPath": "trustStorePasswordPath",
+ "dmaap.security.keyStorePath": "keyStorePath",
+ "dmaap.security.keyStorePasswordPath": "change it",
+ "dmaap.security.enableDmaapCertAuth": "false",
+ "dmaap.dmaapProducerConfiguration": [
+ {
+ "changeIdentifier": "PM_MEAS_FILES",
+ "feedName": "feed03"
+ },
+ {
+ "changeIdentifier": "CTR_MEAS_FILES",
+ "feedName": "feed03"
+ }
+ ],
+ "streams_subscribes": {
+ "dmaap_subscriber": {
+ "dmmap_info": {
+ "topic_url": "http://dradmin:dradmin@mrsim:2222/events/unauthenticated.VES_NOTIFICATION_OUTPUT/OpenDcae-c12/C12"
+ },
+ "type": "message_router"
+ }
+ }
+}
\ No newline at end of file
diff --git a/test/mocks/datafilecollector-testharness/simulator-group/consul/c13_feed2_CTR.json b/test/mocks/datafilecollector-testharness/simulator-group/consul/c13_feed2_CTR.json
new file mode 100644
index 0000000..c8e199d
--- /dev/null
+++ b/test/mocks/datafilecollector-testharness/simulator-group/consul/c13_feed2_CTR.json
@@ -0,0 +1,23 @@
+{
+ "dmaap.ftpesConfig.keyCert": "config/dfc.jks",
+ "dmaap.ftpesConfig.keyPassword": "secret",
+ "dmaap.ftpesConfig.trustedCa": "config/ftp.jks",
+ "dmaap.ftpesConfig.trustedCaPassword": "secret",
+ "dmaap.security.trustStorePath": "change it",
+ "dmaap.security.trustStorePasswordPath": "trustStorePasswordPath",
+ "dmaap.security.keyStorePath": "keyStorePath",
+ "dmaap.security.keyStorePasswordPath": "change it",
+ "dmaap.security.enableDmaapCertAuth": "false",
+ "dmaap.dmaapProducerConfiguration": {
+ "changeIdentifier": "CTR_MEAS_FILES",
+ "feedName": "feed02"
+ },
+ "streams_subscribes": {
+ "dmaap_subscriber": {
+ "dmmap_info": {
+ "topic_url": "http://dradmin:dradmin@mrsim:2222/events/unauthenticated.VES_NOTIFICATION_OUTPUT/OpenDcae-c13/C13"
+ },
+ "type": "message_router"
+ }
+ }
+}
\ No newline at end of file
diff --git a/test/mocks/datafilecollector-testharness/simulator-group/consul/c14_feed3_LOG.json b/test/mocks/datafilecollector-testharness/simulator-group/consul/c14_feed3_LOG.json
new file mode 100644
index 0000000..1f91a38
--- /dev/null
+++ b/test/mocks/datafilecollector-testharness/simulator-group/consul/c14_feed3_LOG.json
@@ -0,0 +1,23 @@
+{
+ "dmaap.ftpesConfig.keyCert": "config/dfc.jks",
+ "dmaap.ftpesConfig.keyPassword": "secret",
+ "dmaap.ftpesConfig.trustedCa": "config/ftp.jks",
+ "dmaap.ftpesConfig.trustedCaPassword": "secret",
+ "dmaap.security.trustStorePath": "change it",
+ "dmaap.security.trustStorePasswordPath": "trustStorePasswordPath",
+ "dmaap.security.keyStorePath": "keyStorePath",
+ "dmaap.security.keyStorePasswordPath": "change it",
+ "dmaap.security.enableDmaapCertAuth": "false",
+ "dmaap.dmaapProducerConfiguration": {
+ "changeIdentifier": "LOG_FILES",
+ "feedName": "feed03"
+ },
+ "streams_subscribes": {
+ "dmaap_subscriber": {
+ "dmmap_info": {
+ "topic_url": "http://dradmin:dradmin@mrsim:2222/events/unauthenticated.VES_NOTIFICATION_OUTPUT/OpenDcae-c14/C14"
+ },
+ "type": "message_router"
+ }
+ }
+}
\ No newline at end of file
diff --git a/test/mocks/datafilecollector-testharness/simulator-group/consul/c15_feed1_PM_feed4_TEST.json b/test/mocks/datafilecollector-testharness/simulator-group/consul/c15_feed1_PM_feed4_TEST.json
new file mode 100644
index 0000000..acef9b9
--- /dev/null
+++ b/test/mocks/datafilecollector-testharness/simulator-group/consul/c15_feed1_PM_feed4_TEST.json
@@ -0,0 +1,29 @@
+{
+ "dmaap.ftpesConfig.keyCert": "config/dfc.jks",
+ "dmaap.ftpesConfig.keyPassword": "secret",
+ "dmaap.ftpesConfig.trustedCa": "config/ftp.jks",
+ "dmaap.ftpesConfig.trustedCaPassword": "secret",
+ "dmaap.security.trustStorePath": "change it",
+ "dmaap.security.trustStorePasswordPath": "trustStorePasswordPath",
+ "dmaap.security.keyStorePath": "keyStorePath",
+ "dmaap.security.keyStorePasswordPath": "change it",
+ "dmaap.security.enableDmaapCertAuth": "false",
+ "dmaap.dmaapProducerConfiguration": [
+ {
+ "changeIdentifier": "PM_MEAS_FILES",
+ "feedName": "feed01"
+ },
+ {
+ "changeIdentifier": "TEST_FILES",
+ "feedName": "feed04"
+ }
+ ],
+ "streams_subscribes": {
+ "dmaap_subscriber": {
+ "dmmap_info": {
+ "topic_url": "http://dradmin:dradmin@mrsim:2222/events/unauthenticated.VES_NOTIFICATION_OUTPUT/OpenDcae-c15/C15"
+ },
+ "type": "message_router"
+ }
+ }
+}
\ No newline at end of file
diff --git a/test/mocks/datafilecollector-testharness/simulator-group/consul/c16_feed4_TEST_feed5_TEMP.json b/test/mocks/datafilecollector-testharness/simulator-group/consul/c16_feed4_TEST_feed5_TEMP.json
new file mode 100644
index 0000000..e10fe07
--- /dev/null
+++ b/test/mocks/datafilecollector-testharness/simulator-group/consul/c16_feed4_TEST_feed5_TEMP.json
@@ -0,0 +1,29 @@
+{
+ "dmaap.ftpesConfig.keyCert": "config/dfc.jks",
+ "dmaap.ftpesConfig.keyPassword": "secret",
+ "dmaap.ftpesConfig.trustedCa": "config/ftp.jks",
+ "dmaap.ftpesConfig.trustedCaPassword": "secret",
+ "dmaap.security.trustStorePath": "change it",
+ "dmaap.security.trustStorePasswordPath": "trustStorePasswordPath",
+ "dmaap.security.keyStorePath": "keyStorePath",
+ "dmaap.security.keyStorePasswordPath": "change it",
+ "dmaap.security.enableDmaapCertAuth": "false",
+ "dmaap.dmaapProducerConfiguration": [
+ {
+ "changeIdentifier": "TEST_FILES",
+ "feedName": "feed04"
+ },
+ {
+ "changeIdentifier": "TEMP_FILES",
+ "feedName": "feed05"
+ }
+ ],
+ "streams_subscribes": {
+ "dmaap_subscriber": {
+ "dmmap_info": {
+ "topic_url": "http://dradmin:dradmin@mrsim:2222/events/unauthenticated.VES_NOTIFICATION_OUTPUT/OpenDcae-c15/C15"
+ },
+ "type": "message_router"
+ }
+ }
+}
\ No newline at end of file
diff --git a/test/mocks/datafilecollector-testharness/simulator-group/consul/consul/cbs_config.hcl b/test/mocks/datafilecollector-testharness/simulator-group/consul/consul/cbs_config.hcl
new file mode 100644
index 0000000..f540975
--- /dev/null
+++ b/test/mocks/datafilecollector-testharness/simulator-group/consul/consul/cbs_config.hcl
@@ -0,0 +1,13 @@
+#server = true
+#bootstrap = true
+#client_addr = "0.0.0.0"
+
+service {
+ # Name for CBS in consul, env var CONFIG_BINDING_SERVICE
+ # should be passed to dfc app with this value
+ Name = "config-binding-service"
+ # Host name where CBS is running
+ Address = "config-binding-service"
+ # Port number where CBS is running
+ Port = 10000
+}
diff --git a/test/mocks/datafilecollector-testharness/simulator-group/consul/consul/cbs_localhost_config.hcl b/test/mocks/datafilecollector-testharness/simulator-group/consul/consul/cbs_localhost_config.hcl
new file mode 100644
index 0000000..c2d9839
--- /dev/null
+++ b/test/mocks/datafilecollector-testharness/simulator-group/consul/consul/cbs_localhost_config.hcl
@@ -0,0 +1,11 @@
+service {
+ # Name for CBS in consul, env var CONFIG_BINDING_SERVICE
+ # should be passed to dfc app with this value
+ # This is only to be used when contacting cbs via local host
+ # (typicall when dfc is executed as an application without a container)
+ Name = "config-binding-service-localhost"
+ # Host name where CBS is running
+ Address = "localhost"
+ # Port number where CBS is running
+ Port = 10000
+}
\ No newline at end of file
diff --git a/test/mocks/datafilecollector-testharness/simulator-group/consul/dmaap_feed1.json b/test/mocks/datafilecollector-testharness/simulator-group/consul/dmaap_feed1.json
new file mode 100644
index 0000000..e6769d8
--- /dev/null
+++ b/test/mocks/datafilecollector-testharness/simulator-group/consul/dmaap_feed1.json
@@ -0,0 +1,10 @@
+{
+ "feed01": {
+ "username": "user",
+ "log_url": "https://drsim:3907/feedlog/1",
+ "publish_url": "https://drsim:3907/publish/1",
+ "location": "loc00",
+ "password": "password",
+ "publisher_id": "972.360gm"
+ }
+}
\ No newline at end of file
diff --git a/test/mocks/datafilecollector-testharness/simulator-group/consul/dmaap_feed1_2_3_4.json b/test/mocks/datafilecollector-testharness/simulator-group/consul/dmaap_feed1_2_3_4.json
new file mode 100644
index 0000000..aa2e6a0
--- /dev/null
+++ b/test/mocks/datafilecollector-testharness/simulator-group/consul/dmaap_feed1_2_3_4.json
@@ -0,0 +1,34 @@
+{
+ "feed01": {
+ "username": "user",
+ "log_url": "https://drsim:3907/feedlog/1",
+ "publish_url": "https://drsim:3907/publish/1",
+ "location": "loc00",
+ "password": "password",
+ "publisher_id": "972.360gm"
+ },
+ "feed02": {
+ "username": "user",
+ "log_url": "https://drsim:3907/feedlog/2",
+ "publish_url": "https://drsim:3907/publish/2",
+ "location": "loc00",
+ "password": "password",
+ "publisher_id": "972.360gm"
+ },
+ "feed03": {
+ "username": "user",
+ "log_url": "https://drsim:3907/feedlog/3",
+ "publish_url": "https://drsim:3907/publish/3",
+ "location": "loc00",
+ "password": "password",
+ "publisher_id": "972.360gm"
+ },
+ "feed04": {
+ "username": "user",
+ "log_url": "https://drsim:3907/feedlog/4",
+ "publish_url": "https://drsim:3907/publish/4",
+ "location": "loc00",
+ "password": "password",
+ "publisher_id": "972.360gm"
+ }
+}
\ No newline at end of file
diff --git a/test/mocks/datafilecollector-testharness/simulator-group/consul/dmaap_feed1_2_3_4_5.json b/test/mocks/datafilecollector-testharness/simulator-group/consul/dmaap_feed1_2_3_4_5.json
new file mode 100644
index 0000000..6f28f39
--- /dev/null
+++ b/test/mocks/datafilecollector-testharness/simulator-group/consul/dmaap_feed1_2_3_4_5.json
@@ -0,0 +1,42 @@
+{
+ "feed01": {
+ "username": "user",
+ "log_url": "https://drsim:3907/feedlog/1",
+ "publish_url": "https://drsim:3907/publish/1",
+ "location": "loc00",
+ "password": "password",
+ "publisher_id": "972.360gm"
+ },
+ "feed02": {
+ "username": "user",
+ "log_url": "https://drsim:3907/feedlog/2",
+ "publish_url": "https://drsim:3907/publish/2",
+ "location": "loc00",
+ "password": "password",
+ "publisher_id": "972.360gm"
+ },
+ "feed03": {
+ "username": "user",
+ "log_url": "https://drsim:3907/feedlog/3",
+ "publish_url": "https://drsim:3907/publish/3",
+ "location": "loc00",
+ "password": "password",
+ "publisher_id": "972.360gm"
+ },
+ "feed04": {
+ "username": "user",
+ "log_url": "https://drsim:3907/feedlog/4",
+ "publish_url": "https://drsim:3907/publish/4",
+ "location": "loc00",
+ "password": "password",
+ "publisher_id": "972.360gm"
+ },
+ "feed05": {
+ "username": "user",
+ "log_url": "https://drsim:3907/feedlog/5",
+ "publish_url": "https://drsim:3907/publish/4",
+ "location": "loc00",
+ "password": "password",
+ "publisher_id": "972.360gm"
+ }
+}
\ No newline at end of file
diff --git a/test/mocks/datafilecollector-testharness/simulator-group/consul/dmaap_feed2.json b/test/mocks/datafilecollector-testharness/simulator-group/consul/dmaap_feed2.json
new file mode 100644
index 0000000..bea360c
--- /dev/null
+++ b/test/mocks/datafilecollector-testharness/simulator-group/consul/dmaap_feed2.json
@@ -0,0 +1,10 @@
+{
+ "feed02": {
+ "username": "user",
+ "log_url": "https://drsim:3907/feedlog/2",
+ "publish_url": "https://drsim:3907/publish/2",
+ "location": "loc00",
+ "password": "password",
+ "publisher_id": "972.360gm"
+ }
+}
\ No newline at end of file
diff --git a/test/mocks/datafilecollector-testharness/simulator-group/consul/dmaap_feed2_3.json b/test/mocks/datafilecollector-testharness/simulator-group/consul/dmaap_feed2_3.json
new file mode 100644
index 0000000..a84bf33
--- /dev/null
+++ b/test/mocks/datafilecollector-testharness/simulator-group/consul/dmaap_feed2_3.json
@@ -0,0 +1,18 @@
+{
+ "feed02": {
+ "username": "user",
+ "log_url": "https://drsim:3907/feedlog/2",
+ "publish_url": "https://drsim:3907/publish/2",
+ "location": "loc00",
+ "password": "password",
+ "publisher_id": "972.360gm"
+ },
+ "feed03": {
+ "username": "user",
+ "log_url": "https://drsim:3907/feedlog/3",
+ "publish_url": "https://drsim:3907/publish/3",
+ "location": "loc00",
+ "password": "password",
+ "publisher_id": "972.360gm"
+ }
+}
\ No newline at end of file
diff --git a/test/mocks/datafilecollector-testharness/simulator-group/consul/dmaap_feed3.json b/test/mocks/datafilecollector-testharness/simulator-group/consul/dmaap_feed3.json
new file mode 100644
index 0000000..11138e3
--- /dev/null
+++ b/test/mocks/datafilecollector-testharness/simulator-group/consul/dmaap_feed3.json
@@ -0,0 +1,10 @@
+{
+ "feed03": {
+ "username": "user",
+ "log_url": "https://drsim:3907/feedlog/3",
+ "publish_url": "https://drsim:3907/publish/3",
+ "location": "loc00",
+ "password": "password",
+ "publisher_id": "972.360gm"
+ }
+}
\ No newline at end of file
diff --git a/test/mocks/datafilecollector-testharness/simulator-group/consul_config.sh b/test/mocks/datafilecollector-testharness/simulator-group/consul_config.sh
new file mode 100755
index 0000000..a3492b9
--- /dev/null
+++ b/test/mocks/datafilecollector-testharness/simulator-group/consul_config.sh
@@ -0,0 +1,43 @@
+#!/bin/bash
+
+# Script to configure consul with json configuration files with 'localhost' urls. This
+# is needed when running the simulator as as a stand-alone app or via a dfc container in 'host' network mode.
+# Assuming the input json files hostnames for MR and DR simulators are given as 'mrsim'/'drsim'
+# See available consul files in the consul dir
+# The script stores a json config for 'dfc_app'<dfc-instance-id>' if arg 'app' is given.
+# And for 'dfc_app'<dfc-instance-id>':dmaap' if arg 'dmaap' is given.
+# Instance id shall be and integer in the range 0..5
+
+. ../common/test_env.sh
+
+if [ $# != 3 ]; then
+ echo "Script needs three args, app|dmaap <dfc-instance-id> <json-file-path>"
+ exit 1
+fi
+
+if [ $2 -lt 0 ] || [ $2 -gt $DFC_MAX_IDX ]; then
+ __print_err "dfc-instance-id should be 0.."$DFC_MAX_IDX
+ exit 1
+fi
+if ! [ -f $3 ]; then
+ __print_err "json file does not extis: "$3
+ exit 1
+fi
+
+if [ $1 == "app" ]; then
+ appname=$DFC_APP_BASE$2
+ echo "Replacing 'mrsim' with 'localhost' in json app config for consul"
+ sed 's/mrsim/localhost/g' $3 > .tmp_file.json
+elif [ $1 == "dmaap" ]; then
+ appname=$DFC_APP_BASE$2":dmaap"
+ echo "Replacing 'drsim' with 'localhost' in json dmaap config for consul"
+ sed 's/drsim/localhost/g' $3 > .tmp_file.json
+else
+ __print_err "config type should be 'app' or 'dmaap'"
+ exit 1
+fi
+
+echo "Configuring consul for " $appname " from " $3
+curl -s http://127.0.0.1:${CONSUL_PORT}/v1/kv/${appname}?dc=dc1 -X PUT -H 'Accept: application/json' -H 'Content-Type: application/json' -H 'X-Requested-With: XMLHttpRequest' --data-binary "@"$tmp_file.json >/dev/null
+
+echo "done"
\ No newline at end of file
diff --git a/test/mocks/datafilecollector-testharness/simulator-group/dfc-internal-stats.sh b/test/mocks/datafilecollector-testharness/simulator-group/dfc-internal-stats.sh
new file mode 100755
index 0000000..9fbe961
--- /dev/null
+++ b/test/mocks/datafilecollector-testharness/simulator-group/dfc-internal-stats.sh
@@ -0,0 +1,103 @@
+#!/bin/bash
+
+# Script to print internal dfc stats every 5 sec to screen and file
+# Default port is 8100 for DFC
+# Useage: ./dfc-internal-stats.sh all|internal|jvm [<dfc-port-number>]
+
+print_usage() {
+ echo "Useage: ./dfc-internal-stats.sh all|internal|jvm [<dfc-port-number>]"
+}
+stat=""
+if [ $# -eq 0 ]; then
+ dfcport=8100
+ stat="all"
+elif [ $# -eq 1 ]; then
+ dfcport=8100
+ stat=$1
+elif [ $# -eq 2 ]; then
+ dfcport=$2
+ stat=$1
+else
+ print_usage
+ exit 1
+fi
+
+heading=1
+
+if [ $stat == "all" ]; then
+ echo "Printing stats for both JVM and DFC using port "$dfcport
+elif [ $stat == "internal" ]; then
+ echo "Printing stats for DFC using port "$dfcport
+elif [ $stat == "jvm" ]; then
+ echo "Printing stats for JVM using port "$dfcport
+else
+ print_usage
+ exit 1
+fi
+fileoutput="./.tmp_stats.txt"
+
+echo "Stats piped to file: "$fileoutput
+
+rm $fileoutput
+
+
+
+floatToInt() {
+ printf "%.0f\n" "$@"
+}
+
+do_curl_actuator() {
+ val=$(curl -s localhost:${dfcport}/actuator/metrics/${1} | grep -o -E "\"value\":[0-9.E]+" | awk -F\: '{print $2}')
+ val=$(floatToInt $val)
+ printf "%-20s %+15s\n" $1 $val
+ if [ $heading -eq 1 ]; then
+ echo -n "," $1 >> $fileoutput
+ else
+ echo -n "," $val >> $fileoutput
+ fi
+}
+
+do_curl_status() {
+ curl -s localhost:${dfcport}/status > ./.tmp_curl_res
+ cat ./.tmp_curl_res
+ while read line; do
+ len=${#line}
+ if [ $len -gt 0 ]; then
+ val=${line#*:}
+ id=${line%"$val"}
+ if [ $heading -eq 1 ]; then
+ echo -n "," $id >> $fileoutput
+ else
+ echo -n "," $val >> $fileoutput
+ fi
+ fi
+ done < ./.tmp_curl_res
+
+}
+
+
+while [ true ]; do
+ if [ $heading -eq 1 ]; then
+ echo -n "date" >> $fileoutput
+ else
+ ds=$(date)
+ echo -n $ds >> $fileoutput
+ fi
+ if [ $stat == "all" ] || [ $stat == "jvm" ]; then
+ echo "========= DFC JVM Stats ========="
+ do_curl_actuator jvm.threads.live
+ do_curl_actuator jvm.threads.peak
+ do_curl_actuator process.files.open
+ do_curl_actuator process.files.max
+ do_curl_actuator jvm.memory.used
+ do_curl_actuator jvm.memory.max
+ fi
+
+ if [ $stat == "all" ] || [ $stat == "internal" ]; then
+ echo "========= DFC internal Stats ========="
+ do_curl_status
+ fi
+ echo "" >> $fileoutput
+ heading=0
+ sleep 5
+done
\ No newline at end of file
diff --git a/test/mocks/datafilecollector-testharness/simulator-group/docker-compose-setup.sh b/test/mocks/datafilecollector-testharness/simulator-group/docker-compose-setup.sh
index b9b38f8..b212fc2 100755
--- a/test/mocks/datafilecollector-testharness/simulator-group/docker-compose-setup.sh
+++ b/test/mocks/datafilecollector-testharness/simulator-group/docker-compose-setup.sh
@@ -1,14 +1,28 @@
#/bin/bash
-#Script for manually starting all simulators with test setting below
+# Script for manually starting all simulators with test setting below
+# Matching json config is needed in CBS/Consul as well. Use consul_config.sh to add config to consul
-export DR_TC="--tc normal" #Test behaviour for DR sim
-export DR_REDIR_TC="--tc normal" #Test behaviour for DR redir sim
-export MR_TC="--tc710" #Test behaviour for MR sim
-export BC_TC="" #Not in use yet
-export NUM_FTPFILES="105" #Number of FTP files to generate per PNF
-export NUM_PNFS="700" #Number of unuqie PNFs to generate FTP file for
-export FILE_SIZE="1MB" #File size for FTP file (1KB, 1MB, 5MB, 50MB or ALL)
-export FTP_TYPE="SFTP" #Type of FTP files to generate (SFTP, FTPS or ALL)
+export MR_TC="--tc710" # Test behaviour for MR sim
+export MR_GROUPS="OpenDcae-c12:PM_MEAS_FILES" # Comma-separated list of <consumer-group>:<change-identifier>
+export MR_FILE_PREFIX_MAPPING="PM_MEAS_FILES:A" # Comma-separated list of <change-identifer>:<file-name-prefix>
-source ./simulators-start.sh
\ No newline at end of file
+export DR_TC="--tc normal" # Test behaviour for DR sim
+export DR_FEEDS="1:A,2:B,3:C,4:D" # Comma-separated of <feed-id>:<file-name-prefixes> for DR sim
+
+export DR_REDIR_TC="--tc normal" # Test behaviour for DR redir sim
+export DR_REDIR_FEEDS="1:A,2:B,3:C,4:D" # Comma-separated of <feed-id>:<file-name-prefixes> for DR redir sim
+
+export NUM_FTPFILES="105" # Number of FTP files to generate per PNF
+export NUM_PNFS="700" # Number of unuqie PNFs to generate FTP file for
+export FILE_SIZE="1MB" # File size for FTP file (1KB, 1MB, 5MB, 50MB or ALL)
+export FTP_TYPE="SFTP" # Type of FTP files to generate (SFTP, FTPS or ALL)
+export FTP_FILE_PREFIXES="A,B,C,D" # Comma separated list of file name prefixes for ftp files
+export NUM_FTP_SERVERS=1 # Number of FTP server to distribute the PNFs (Max 5)
+
+export SFTP_SIMS="localhost:21,localhost:22,localhost:23,localhost:24,localhost:25" # Comma separated list for SFTP servers host:port
+export FTPS_SIMS="localhost:1022,localhost:1023,localhost:1024,localhost:1026,localhost:1026" # Comma separated list for FTPS servers host:port
+
+export DR_REDIR_SIM="localhost" # Hostname of DR redirect server
+
+source ./simulators-start.sh
diff --git a/test/mocks/datafilecollector-testharness/simulator-group/docker-compose-template.yml b/test/mocks/datafilecollector-testharness/simulator-group/docker-compose-template.yml
index 8505631..f078d36 100644
--- a/test/mocks/datafilecollector-testharness/simulator-group/docker-compose-template.yml
+++ b/test/mocks/datafilecollector-testharness/simulator-group/docker-compose-template.yml
@@ -1,17 +1,40 @@
version: '2'
networks:
- dfcnet:
- ipam:
- config:
- - subnet: 192.168.100.0/16
-
+ dfcnet:
+ external:
+ name: dfcnet
+
services:
-
+
+ consul-server:
+ networks:
+ - dfcnet
+ container_name: dfc_consul
+ image: docker.io/consul:1.4.4
+ ports:
+ - "8500:8500"
+ volumes:
+ - ./consul/consul/:/consul/config
+
+ config-binding-service:
+ networks:
+ - dfcnet
+ container_name: dfc_cbs
+ image: nexus3.onap.org:10001/onap/org.onap.dcaegen2.platform.configbinding.app-app:2.3.0
+ ports:
+ - "10000:10000"
+ environment:
+ - CONSUL_HOST=consul-server
+ depends_on:
+ - consul-server
+
drsim:
networks:
- dfcnet:
- ipv4_address: 192.168.100.2
+ - dfcnet
+ environment:
+ DRR_SIM_IP: ${DR_REDIR_SIM}
+ DR_FEEDS: ${DR_FEEDS}
image: drsim_common:latest
ports:
- "3906:3906"
@@ -21,8 +44,10 @@
drsim_redir:
networks:
- dfcnet:
- ipv4_address: 192.168.100.3
+ - dfcnet
+ environment:
+ DR_SIM_IP: drsim
+ DR_REDIR_FEEDS: ${DR_REDIR_FEEDS}
image: drsim_common:latest
ports:
- "3908:3908"
@@ -32,42 +57,131 @@
mrsim:
networks:
- dfcnet:
- ipv4_address: 192.168.100.1
+ - dfcnet
+ environment:
+ SFTP_SIMS: ${SFTP_SIMS}
+ FTPS_SIMS: ${FTPS_SIMS}
+ NUM_FTP_SERVERS: ${NUM_FTP_SERVERS}
+ MR_GROUPS: ${MR_GROUPS}
+ MR_FILE_PREFIX_MAPPING: ${MR_FILE_PREFIX_MAPPING}
image: mrsim:latest
ports:
- "2222:2222"
container_name: dfc_mr-sim
command: python mr-sim.py ${MR_TC}
- sftp-server:
- network_mode: bridge
- container_name: dfc_sftp-server
+ sftp-server0:
+ networks:
+ - dfcnet
+ container_name: dfc_sftp-server0
image: atmoz/sftp:alpine
ports:
- "1022:22"
restart: on-failure
command: onap:pano:1001
-
- ftpes-server-vsftpd:
- network_mode: bridge
- container_name: dfc_ftpes-server-vsftpd
- image: docker.io/panubo/vsftpd
+ sftp-server1:
+ networks:
+ - dfcnet
+ container_name: dfc_sftp-server1
+ image: atmoz/sftp:alpine
ports:
- - "21:21"
- - "8001-8010:8001-8010"
+ - "1023:22"
+ restart: on-failure
+ command: onap:pano:1001
+
+ sftp-server2:
+ networks:
+ - dfcnet
+ container_name: dfc_sftp-server2
+ image: atmoz/sftp:alpine
+ ports:
+ - "1024:22"
+ restart: on-failure
+ command: onap:pano:1001
+
+ sftp-server3:
+ networks:
+ - dfcnet
+ container_name: dfc_sftp-server3
+ image: atmoz/sftp:alpine
+ ports:
+ - "1025:22"
+ restart: on-failure
+ command: onap:pano:1001
+
+ sftp-server4:
+ networks:
+ - dfcnet
+ container_name: dfc_sftp-server4
+ image: atmoz/sftp:alpine
+ ports:
+ - "1026:22"
+ restart: on-failure
+ command: onap:pano:1001
+
+ ftpes-server-vsftpd0:
+ networks:
+ - dfcnet
+ container_name: dfc_ftpes-server-vsftpd0
+ image: ftps_vsftpd:latest
+ ports:
+ - "1032:21"
environment:
FTP_USER: onap
FTP_PASSWORD: pano
- PASV_ADDRESS: localhost
- PASV_MIN_PORT: 8001
- PASV_MAX_PORT: 8010
- volumes:
- - ./tls/ftp.crt:/etc/ssl/private/ftp.crt:ro
- - ./tls/ftp.key:/etc/ssl/private/ftp.key:ro
- - ./tls/dfc.crt:/etc/ssl/private/dfc.crt:ro
- - ./configuration/vsftpd_ssl.conf:/etc/vsftpd_ssl.conf:ro
-
restart: on-failure
command: vsftpd /etc/vsftpd_ssl.conf
+
+ ftpes-server-vsftpd1:
+ networks:
+ - dfcnet
+ container_name: dfc_ftpes-server-vsftpd1
+ image: ftps_vsftpd:latest
+ ports:
+ - "1033:21"
+ environment:
+ FTP_USER: onap
+ FTP_PASSWORD: pano
+ restart: on-failure
+ command: vsftpd /etc/vsftpd_ssl.conf
+
+ ftpes-server-vsftpd2:
+ networks:
+ - dfcnet
+ container_name: dfc_ftpes-server-vsftpd2
+ image: ftps_vsftpd:latest
+ ports:
+ - "1034:21"
+ environment:
+ FTP_USER: onap
+ FTP_PASSWORD: pano
+ restart: on-failure
+ command: vsftpd /etc/vsftpd_ssl.conf
+
+ ftpes-server-vsftpd3:
+ networks:
+ - dfcnet
+ container_name: dfc_ftpes-server-vsftpd3
+ image: ftps_vsftpd:latest
+ ports:
+ - "1035:21"
+ environment:
+ FTP_USER: onap
+ FTP_PASSWORD: pano
+ restart: on-failure
+ command: vsftpd /etc/vsftpd_ssl.conf
+
+ ftpes-server-vsftpd4:
+ networks:
+ - dfcnet
+ container_name: dfc_ftpes-server-vsftpd4
+ image: ftps_vsftpd:latest
+ ports:
+ - "1036:21"
+ environment:
+ FTP_USER: onap
+ FTP_PASSWORD: pano
+ restart: on-failure
+ command: vsftpd /etc/vsftpd_ssl.conf
+
diff --git a/test/mocks/datafilecollector-testharness/simulator-group/prepare-images.sh b/test/mocks/datafilecollector-testharness/simulator-group/prepare-images.sh
new file mode 100755
index 0000000..666e14a
--- /dev/null
+++ b/test/mocks/datafilecollector-testharness/simulator-group/prepare-images.sh
@@ -0,0 +1,19 @@
+#/bin/bash
+
+#Script for manually preparing images for mr-sim, dr-sim, dr-redir-sim and sftp server.
+
+#Build MR sim
+cd ../mr-sim
+
+docker build -t mrsim:latest .
+
+#Build DR sim common image
+cd ../dr-sim
+
+docker build -t drsim_common:latest .
+
+#Build image for ftps server
+cd ../ftps-sftp-server
+
+docker build -t ftps_vsftpd:latest -f Dockerfile-ftps .
+
diff --git a/test/mocks/datafilecollector-testharness/simulator-group/setup-ftp-files-for-image.sh b/test/mocks/datafilecollector-testharness/simulator-group/setup-ftp-files-for-image.sh
index 7685c81..b1fa01e 100755
--- a/test/mocks/datafilecollector-testharness/simulator-group/setup-ftp-files-for-image.sh
+++ b/test/mocks/datafilecollector-testharness/simulator-group/setup-ftp-files-for-image.sh
@@ -4,44 +4,61 @@
# The file names matches the files names in the events polled from the MR simulator.
# Intended for execution in the running ftp containers in the ftp-root dir.
-NUM=200 #Default number of files
+NUM=200 #Default number of files
PNFS=1 #Default number of PNFs
FSIZE="ALL"
+PREFIXES="A"
+FTP_SERV_INDEX=0
+NUM_FTP_SERVERS=1
-if [ $# -eq 1 ]; then
+if [ $# -ge 1 ]; then
NUM=$1
-elif [ $# -eq 2 ]; then
- NUM=$1
+fi
+if [ $# -ge 2 ]; then
PNFS=$2
-elif [ $# -eq 3 ]; then
- NUM=$1
- PNFS=$2
+fi
+if [ $# -ge 3 ]; then
FSIZE=$3
if [ $3 != "1KB" ] && [ $3 != "1MB" ] && [ $3 != "5MB" ] && [ $3 != "50MB" ] && [ $3 != "ALL" ]; then
echo "File size shall be 1KB|1MB|5MB|50MB|ALL"
exit
fi
-else
- echo "Wrong args, usage: setup-ftp-files-for-image.sh [ <num-files> [ <num-PNFs> [ 1KB|1MB|5MB|50MB ] ] ]"
+fi
+if [ $# -ge 4 ]; then
+ PREFIXES=$4
+fi
+if [ $# -ge 5 ]; then
+ NUM_FTP_SERVERS=$5
+fi
+if [ $# -ge 6 ]; then
+ FTP_SERV_INDEX=$6
+fi
+if [ $# -lt 1 ] || [ $# -gt 6 ]; then
+ echo "Wrong args, usage: setup-ftp-files-for-image.sh [ <num-files> [ <num-PNFs> [ 1KB|1MB|5MB|50MB [ <comma-separated-file-name-prefixs> [ <number-of-ftp-servers> <ftp-server-index> ] ] ] ] ] ]"
exit
fi
-echo "Running ftp file creations. " $PNFS " PNFs and " $NUM " files for each PNF with file size(s) "$FSIZE
+echo "Running ftp file creations. " $PNFS " PNFs and " $NUM " files for each PNF with file size(s) " $FSIZE "and file prefixe(s) " $PREFIXES " in ftp servers with index " $FTP_SERV_INDEX
truncate -s 1KB 1KB.tar.gz
truncate -s 1MB 1MB.tar.gz
truncate -s 5MB 5MB.tar.gz
truncate -s 50MB 50MB.tar.gz
-p=0
-while [ $p -lt $PNFS ]; do
- i=0
- while [ $i -lt $NUM ]; do #Problem with for loop and var substituion in curly bracket....so used good old style loop
- if [ $FSIZE = "ALL" ] || [ $FSIZE = "1KB" ]; then ln -s 1KB.tar.gz 'A20000626.2315+0200-2330+0200_PNF'$p'-'$i'-1KB.tar.gz' >& /dev/null; fi
- if [ $FSIZE = "ALL" ] || [ $FSIZE = "1MB" ]; then ln -s 1MB.tar.gz 'A20000626.2315+0200-2330+0200_PNF'$p'-'$i'-1MB.tar.gz' >& /dev/null; fi
- if [ $FSIZE = "ALL" ] || [ $FSIZE = "5MB" ]; then ln -s 5MB.tar.gz 'A20000626.2315+0200-2330+0200_PNF'$p'-'$i'-5MB.tar.gz' >& /dev/null; fi
- if [ $FSIZE = "ALL" ] || [ $FSIZE = "50MB" ]; then ln -s 50MB.tar.gz 'A20000626.2315+0200-2330+0200_PNF'$p'-'$i'-50MB.tar.gz' >& /dev/null; fi
- let i=i+1
- done
- let p=p+1
+for fnp in ${PREFIXES//,/ }
+do
+ p=0
+ while [ $p -lt $PNFS ]; do
+ if [[ $(($p%$NUM_FTP_SERVERS)) == $FTP_SERV_INDEX ]]; then
+ i=0
+ while [ $i -lt $NUM ]; do #Problem with for loop and var substituion in curly bracket....so used good old style loop
+ if [ $FSIZE = "ALL" ] || [ $FSIZE = "1KB" ]; then ln -s 1KB.tar.gz $fnp'20000626.2315+0200-2330+0200_PNF'$p'-'$i'-1KB.tar.gz' >& /dev/null; fi
+ if [ $FSIZE = "ALL" ] || [ $FSIZE = "1MB" ]; then ln -s 1MB.tar.gz $fnp'20000626.2315+0200-2330+0200_PNF'$p'-'$i'-1MB.tar.gz' >& /dev/null; fi
+ if [ $FSIZE = "ALL" ] || [ $FSIZE = "5MB" ]; then ln -s 5MB.tar.gz $fnp'20000626.2315+0200-2330+0200_PNF'$p'-'$i'-5MB.tar.gz' >& /dev/null; fi
+ if [ $FSIZE = "ALL" ] || [ $FSIZE = "50MB" ]; then ln -s 50MB.tar.gz $fnp'20000626.2315+0200-2330+0200_PNF'$p'-'$i'-50MB.tar.gz' >& /dev/null; fi
+ let i=i+1
+ done
+ fi
+ let p=p+1
+ done
done
diff --git a/test/mocks/datafilecollector-testharness/simulator-group/sim-monitor-start.sh b/test/mocks/datafilecollector-testharness/simulator-group/sim-monitor-start.sh
index 52c8c1c..79aab90 100755
--- a/test/mocks/datafilecollector-testharness/simulator-group/sim-monitor-start.sh
+++ b/test/mocks/datafilecollector-testharness/simulator-group/sim-monitor-start.sh
@@ -1,7 +1,7 @@
#/bin/bash
-#Script to start the sim-monitor
+#Script to install dependencies and start the sim-monitor
-#Re-using modules for dr-sim
-cp -r ../dr-sim/node_modules .
-node sim-monitor.js
\ No newline at end of file
+npm install express
+node sim-monitor
+
diff --git a/test/mocks/datafilecollector-testharness/simulator-group/sim-monitor.js b/test/mocks/datafilecollector-testharness/simulator-group/sim-monitor.js
index 634d144..e15b637 100644
--- a/test/mocks/datafilecollector-testharness/simulator-group/sim-monitor.js
+++ b/test/mocks/datafilecollector-testharness/simulator-group/sim-monitor.js
@@ -1,7 +1,14 @@
+// Sim mon server - query the simulators for counters and other data
+// Presents a web page on localhost:9999/mon
+
var http = require('http');
var express = require('express');
var app = express();
+var fieldSize=32;
+
+var dfcHeadings=[];
+var dfcVal=[];
//I am alive
app.get("/",function(req, res){
@@ -19,12 +26,6 @@
// The whole response has been received.
resp.on('end', () => {
- //Pad data to fixed length
- var i = 20-data.length;
- while(i>0) {
- data = data+" ";
- i--;
- }
cb(data);
});
@@ -34,75 +35,203 @@
});
};
+
+//Format a comma separated list of data to a html-safe string with fixed fieldsizes
+function formatDataRow(commaList) {
+ var str = "";
+ var tmp=commaList.split(',');
+ for(i=0;i<tmp.length;i++) {
+ data=tmp[i];
+ var len = fieldSize-data.length;
+ while(len>0) {
+ data = data+" ";
+ len--;
+ }
+ str=str+data+" ";
+ }
+ return str;
+}
+
+//Format a comma separated list of ids to a html-safe string with fixed fieldsizes
+function formatIdRow(commaList) {
+ var str = "";
+ var tmp=commaList.split(',');
+ for(i=0;i<tmp.length;i++) {
+ tmp[i] = tmp[i].trim();
+ data="<"+tmp[i]+">";
+ var len = fieldSize+4-data.length;
+ while(len>0) {
+ data = data+" ";
+ len--;
+ }
+ str=str+data+" ";
+ }
+ return str;
+}
+
+//Format a list of ids to a html-safe string in compact format
+function formatIdRowCompact(commaList) {
+ var str = "";
+ var tmp=commaList.split(',');
+ for(i=0;i<tmp.length;i++) {
+ tmp[i] = tmp[i].trim();
+ data="<"+tmp[i]+">";
+ str=str+data+" ";
+ }
+ return str;
+}
+
+function buildDfcData(dfc, idx) {
+
+ if (dfcHeadings.length == 0) {
+ dfcVal[0]=[];
+ dfcVal[1]=[];
+ dfcVal[2]=[];
+ dfcVal[3]=[];
+ dfcVal[4]=[];
+ if (dfc.indexOf("no response") > -1) {
+ return;
+ } else {
+ dfc=dfc.replace(/\n/g, " ");
+ dfc=dfc.replace(/\r/g, " ");
+ var tmp=dfc.split(' ');
+ var ctr=0
+ for(i=0;i<tmp.length;i++) {
+ tmp[i]=tmp[i].trim();
+ if (tmp[i].length>0) {
+ if (ctr%2==0) {
+ dfcHeadings[ctr/2]=tmp[i];
+ }
+ ctr=ctr+1;
+ }
+ }
+ }
+ }
+ if (dfcHeadings.length > 0) {
+ if (dfc.indexOf("no response") > -1) {
+ dfcVal[idx]=[];
+ return;
+ } else {
+ dfc=dfc.replace(/\n/g, " ");
+ dfc=dfc.replace(/\r/g, " ");
+ var tmp=dfc.split(' ');
+ var ctr=0
+ for(i=0;i<tmp.length;i++) {
+ tmp[i]=tmp[i].trim();
+ if (tmp[i].length>0) {
+ if (ctr%2==1) {
+ dfcVal[idx][Math.trunc(ctr/2)]=""+tmp[i];
+ }
+ ctr=ctr+1;
+ }
+ }
+ }
+ }
+}
+
+function padding(val, fieldSize, pad) {
+ s=""+val;
+ for(i=s.length;i<fieldSize;i++) {
+ s=s+pad
+ }
+ return s;
+}
+
//Status variables, for parameters values fetched from other simulators
-var mr1, mr2, mr3, mr4, mr5, mr6, mr7, mr8, mr9, mr10;
+var mr1="", mr2="", mr3="", mr4="", mr5="", mr6="", mr7="", mr8="", mr9="", mr10="", mr11="", mr12="", mr13="";
-var dr1, dr2, dr3, dr4, dr5, dr6, dr7, dr8, dr9, dr10;
+var dr1="", dr2="", dr3="", dr4="", dr5="", dr6="", dr7="", dr8="", dr9="", dr10="", dr11="", dr12="", dr13="";
-var drr1, drr2, drr3, drr4, drr5, drr6;
+var drr1="", drr2="", drr3="", drr4="", drr5="", drr6="", drr7="", drr8="", drr9="";
//Heartbeat var
-var dfc1;
+var dfc0,dfc1,dfc2,dfc3,dfc4;
app.get("/mon",function(req, res){
//DFC
- getSimCtr("http://127.0.0.1:8100/heartbeat", function(data) {
+ getSimCtr("http://127.0.0.1:8100/status", function(data) {
+ dfc0 = data;
+ buildDfcData(dfc0, 0);
+ });
+ getSimCtr("http://127.0.0.1:8101/status", function(data) {
dfc1 = data;
+ buildDfcData(dfc1, 1);
+ });
+ getSimCtr("http://127.0.0.1:8102/status", function(data) {
+ dfc2 = data;
+ buildDfcData(dfc2, 2);
+ });
+ getSimCtr("http://127.0.0.1:8103/status", function(data) {
+ dfc3 = data;
+ buildDfcData(dfc3, 3);
+ });
+ getSimCtr("http://127.0.0.1:8104/status", function(data) {
+ dfc4 = data;
+ buildDfcData(dfc4, 4);
});
//MR
- getSimCtr("http://127.0.0.1:2222/ctr_requests", function(data) {
+ getSimCtr("http://127.0.0.1:2222/groups/ctr_requests", function(data) {
mr1 = data;
});
- getSimCtr("http://127.0.0.1:2222/ctr_responses", function(data) {
+ getSimCtr("http://127.0.0.1:2222/groups/ctr_responses", function(data) {
mr2 = data;
});
- getSimCtr("http://127.0.0.1:2222/ctr_unique_files", function(data) {
+ getSimCtr("http://127.0.0.1:2222/groups/ctr_unique_files", function(data) {
mr3 = data;
});
getSimCtr("http://127.0.0.1:2222/tc_info", function(data) {
mr4 = data;
});
- getSimCtr("http://127.0.0.1:2222/ctr_events", function(data) {
+ getSimCtr("http://127.0.0.1:2222/groups/ctr_events", function(data) {
mr5 = data;
});
getSimCtr("http://127.0.0.1:2222/execution_time", function(data) {
mr6 = data;
});
- getSimCtr("http://127.0.0.1:2222/ctr_unique_PNFs", function(data) {
+ getSimCtr("http://127.0.0.1:2222/groups/ctr_unique_PNFs", function(data) {
mr7 = data;
});
- getSimCtr("http://127.0.0.1:2222/exe_time_first_poll", function(data) {
+ getSimCtr("http://127.0.0.1:2222/groups/exe_time_first_poll", function(data) {
mr8 = data;
});
- getSimCtr("http://127.0.0.1:2222/ctr_files", function(data) {
+ getSimCtr("http://127.0.0.1:2222/groups/ctr_files", function(data) {
mr9 = data;
});
getSimCtr("http://127.0.0.1:2222/status", function(data) {
mr10 = data;
});
+ getSimCtr("http://127.0.0.1:2222/groups", function(data) {
+ mr11 = data;
+ });
+ getSimCtr("http://127.0.0.1:2222/changeids", function(data) {
+ mr12 = data;
+ });
+ getSimCtr("http://127.0.0.1:2222/fileprefixes", function(data) {
+ mr13 = data;
+ });
//DR
- getSimCtr("http://127.0.0.1:3906/ctr_publish_query", function(data) {
+ getSimCtr("http://127.0.0.1:3906/feeds/ctr_publish_query", function(data) {
dr1 = data;
});
- getSimCtr("http://127.0.0.1:3906/ctr_publish_query_published", function(data) {
+ getSimCtr("http://127.0.0.1:3906/feeds/ctr_publish_query_published", function(data) {
dr2 = data;
- });
- getSimCtr("http://127.0.0.1:3906/ctr_publish_query_not_published", function(data) {
+ });
+ getSimCtr("http://127.0.0.1:3906/feeds/ctr_publish_query_not_published", function(data) {
dr3 = data;
});
- getSimCtr("http://127.0.0.1:3906/ctr_publish_req", function(data) {
+ getSimCtr("http://127.0.0.1:3906/feeds/ctr_publish_req", function(data) {
dr4 = data;
});
- getSimCtr("http://127.0.0.1:3906/ctr_publish_req_redirect", function(data) {
+ getSimCtr("http://127.0.0.1:3906/feeds/ctr_publish_req_redirect", function(data) {
dr5 = data;
});
- getSimCtr("http://127.0.0.1:3906/ctr_publish_req_published", function(data) {
+ getSimCtr("http://127.0.0.1:3906/feeds/ctr_publish_req_published", function(data) {
dr6 = data;
});
- getSimCtr("http://127.0.0.1:3906/ctr_published_files", function(data) {
+ getSimCtr("http://127.0.0.1:3906/feeds/ctr_published_files", function(data) {
dr7 = data;
});
getSimCtr("http://127.0.0.1:3906/tc_info", function(data) {
@@ -111,15 +240,24 @@
getSimCtr("http://127.0.0.1:3906/execution_time", function(data) {
dr9 = data;
});
- getSimCtr("http://127.0.0.1:3906/ctr_double_publish", function(data) {
+ getSimCtr("http://127.0.0.1:3906/feeds/ctr_double_publish", function(data) {
dr10 = data;
});
+ getSimCtr("http://127.0.0.1:3906/feeds", function(data) {
+ dr11=data;
+ });
+ getSimCtr("http://127.0.0.1:3906/feeds/ctr_publish_query_bad_file_prefix", function(data) {
+ dr12=data;
+ });
+ getSimCtr("http://127.0.0.1:3906/feeds/ctr_publish_req_bad_file_prefix",function(data) {
+ dr13=data;
+ });
//DR REDIR
- getSimCtr("http://127.0.0.1:3908/ctr_publish_requests", function(data) {
+ getSimCtr("http://127.0.0.1:3908/feeds/ctr_publish_requests", function(data) {
drr1 = data;
});
- getSimCtr("http://127.0.0.1:3908/ctr_publish_responses", function(data) {
+ getSimCtr("http://127.0.0.1:3908/feeds/ctr_publish_responses", function(data) {
drr2 = data;
});
getSimCtr("http://127.0.0.1:3908/tc_info", function(data) {
@@ -128,66 +266,117 @@
getSimCtr("http://127.0.0.1:3908/execution_time", function(data) {
drr4 = data;
});
- getSimCtr("http://127.0.0.1:3908/time_lastpublish", function(data) {
+ getSimCtr("http://127.0.0.1:3908/feeds/time_lastpublish", function(data) {
drr5 = data;
});
- getSimCtr("http://127.0.0.1:3908/dwl_volume", function(data) {
+ getSimCtr("http://127.0.0.1:3908/feeds/dwl_volume", function(data) {
drr6 = data;
});
+ getSimCtr("http://127.0.0.1:3908/feeds", function(data) {
+ drr7=data;
+ });
+ getSimCtr("http://127.0.0.1:3908/feeds/ctr_publish_requests_bad_file_prefix", function(data) {
+ drr8 = data;
+ });
+ getSimCtr("http://127.0.0.1:3908/speed", function(data) {
+ drr9 = data;
+ });
//Build web page
var str = "<!DOCTYPE html>" +
"<html>" +
"<head>" +
- "<meta http-equiv=\"refresh\" content=\"5\">"+ //5 sec auto reefresh
+ "<meta http-equiv=\"refresh\" content=\"5\">"+ //5 sec auto refresh
"<title>DFC and simulator monitor</title>"+
"</head>" +
"<body>" +
- "<h3>DFC</h3>" +
- "<font face=\"Courier New\">"+
- "Heartbeat:....................................." + dfc1 + "<br>" +
- "</font>"+
+ "<h3>DFC apps</h3>" +
+ "<font face=\"monospace\">";
+// "dfc_app0: " + dfc0 + "<br>" +
+// "dfc_app1: " + dfc1 + "<br>" +
+// "dfc_app2: " + dfc2 + "<br>" +
+// "dfc_app3: " + dfc3 + "<br>" +
+// "dfc_app4: " + dfc4 + "<br>";
+
+ for(id=0;id<5;id++) {
+ if (id==0) {
+ str=str+padding("Instance",22,".");
+ str=str+" "+" ";
+ }
+ str=str+padding("dfc_app"+id,26, " ");
+ str=str+" "+" ";
+ }
+ str=str+"<br>";
+
+ if (dfcHeadings.length > 0) {
+ var hl=0;
+ for(hl=0;hl<dfcHeadings.length;hl++) {
+ str=str+padding(dfcHeadings[hl], 22, ".");
+ for(id=0;id<5;id++) {
+ if (dfcVal[id].length > 0) {
+ val=""+padding(dfcVal[id][hl], 26, " ");
+ } else {
+ val=""+padding("-", 26, " ");
+ }
+ str=str+" "+" "+val;
+ }
+ str=str+"<br>";
+ }
+ }
+
+ str=str+"</font>"+
"<h3>MR Simulator</h3>" +
- "<font face=\"Courier New\">"+
+ "<font face=\"monospace\">"+
"MR TC:........................................." + mr4 + "<br>" +
+ "Configured filename prefixes:.................." + formatIdRowCompact(mr13) + "<br>" +
"Status:........................................" + mr10 + "<br>" +
"Execution time (mm.ss):........................" + mr6 + "<br>" +
- "Execution time from first poll (mm.ss):....... " + mr8 + "<br>" +
- "Number of requests (polls):...................." + mr1 + "<br>" +
- "Number of responses (polls):..................." + mr2 + "<br>" +
- "Number of files in all responses:.............." + mr9 + "<br>" +
- "Number of unique files in all responses:......." + mr3 + "<br>" +
- "Number of events..............................." + mr5 + "<br>" +
- "Number of unique PNFs.........................." + mr7 + "<br>" +
+ "Configured groups:............................." + formatIdRow(mr11) + "<br>" +
+ "Configured change identifiers:................." + formatIdRow(mr12) + "<br>" +
+ "Execution time from first poll (mm.ss):....... " + formatDataRow(mr8) + "<br>" +
+ "Number of requests (polls):...................." + formatDataRow(mr1) + "<br>" +
+ "Number of responses (polls):..................." + formatDataRow(mr2) + "<br>" +
+ "Number of files in all responses:.............." + formatDataRow(mr9) + "<br>" +
+ "Number of unique files in all responses:......." + formatDataRow(mr3) + "<br>" +
+ "Number of events..............................." + formatDataRow(mr5) + "<br>" +
+ "Number of unique PNFs.........................." + formatDataRow(mr7) + "<br>" +
"</font>"+
"<h3>DR Simulator</h3>" +
- "<font face=\"Courier New\">"+
+ "<font face=\"monospace\">"+
"DR TC:........................................." + dr8 + "<br>" +
"Execution time (mm.ss):........................" + dr9 + "<br>" +
- "Number of queries:............................." + dr1 + "<br>" +
- "Number of query responses, file published:....." + dr2 + "<br>" +
- "Number of query responses, file not published:." + dr3 + "<br>" +
- "Number of requests:............................" + dr4 + "<br>" +
- "Number of responses with redirect:............." + dr5 + "<br>" +
- "Number of responses without redirect:.........." + dr6 + "<br>" +
- "Number of published files:....................." + dr7 + "<br>" +
- "Number of double published files:.............." + dr10 + "<br>" +
+ "Configured feeds (feedId:filePrefix)..........." + formatIdRow(dr11) +"<br>" +
+ "Number of queries:............................." + formatDataRow(dr1) + "<br>" +
+ "Number of queries with bad file name prefix:..." + formatDataRow(dr12) + "<br>" +
+ "Number of query responses, file published:....." + formatDataRow(dr2) + "<br>" +
+ "Number of query responses, file not published:." + formatDataRow(dr3) + "<br>" +
+ "Number of requests:............................" + formatDataRow(dr4) + "<br>" +
+ "Number of requests with bad file name prefix:.." + formatDataRow(dr13) + "<br>" +
+ "Number of responses with redirect:............." + formatDataRow(dr5) + "<br>" +
+ "Number of responses without redirect:.........." + formatDataRow(dr6) + "<br>" +
+ "Number of published files:....................." + formatDataRow(dr7) + "<br>" +
+ "Number of double published files:.............." + formatDataRow(dr10) + "<br>" +
"</font>"+
"<h3>DR Redirect Simulator</h3>" +
- "<font face=\"Courier New\">"+
+ "<font face=\"monospace\">" +
"DR REDIR TC:..................................." + drr3 + "<br>" +
"Execution time (mm.ss):........................" + drr4 + "<br>" +
- "Number of requests:............................" + drr1 + "<br>" +
- "Number of responses:..........................." + drr2 + "<br>" +
- "Downloaded volume (bytes):....................." + drr6 + "<br>" +
- "Last publish (mm:ss):.........................." + drr5 + "<br>" +
+ "Publish speed (files/sec):....................." + drr9 + "<br>" +
+ "Configured feeds (feedId:filePrefix)..........." + formatIdRow(drr7) +"<br>" +
+ "Number of requests:............................" + formatDataRow(drr1) + "<br>" +
+ "Number of requests with bad file name prefix:.." + formatDataRow(drr8) + "<br>" +
+ "Number of responses:..........................." + formatDataRow(drr2) + "<br>" +
+ "Downloaded volume (bytes):....................." + formatDataRow(drr6) + "<br>" +
+ "Last publish (mm:ss):.........................." + formatDataRow(drr5) + "<br>" +
"</font>"+
"</body>" +
"</html>";
res.send(str);
+
})
var httpServer = http.createServer(app);
var httpPort=9999;
httpServer.listen(httpPort);
-console.log("Simulator monitor listening (http) at "+httpPort);
\ No newline at end of file
+console.log("Simulator monitor listening (http) at "+httpPort);
+console.log("Open the web page on localhost:9999/mon to view the statistics page.")
\ No newline at end of file
diff --git a/test/mocks/datafilecollector-testharness/simulator-group/simulators-kill.sh b/test/mocks/datafilecollector-testharness/simulator-group/simulators-kill.sh
index 3f0ba35..c0526e9 100755
--- a/test/mocks/datafilecollector-testharness/simulator-group/simulators-kill.sh
+++ b/test/mocks/datafilecollector-testharness/simulator-group/simulators-kill.sh
@@ -1,10 +1,39 @@
#!/bin/bash
-#Stop all simulators
+#Script to kill and remove all simulators
+echo "Killing simulator containers"
docker kill dfc_dr-sim
docker kill dfc_dr-redir-sim
docker kill dfc_mr-sim
-docker kill dfc_sftp-server
-docker kill dfc_ftpes-server-vsftpd
+docker kill dfc_sftp-server0
+docker kill dfc_sftp-server1
+docker kill dfc_sftp-server2
+docker kill dfc_sftp-server3
+docker kill dfc_sftp-server4
+docker kill dfc_ftpes-server-vsftpd0
+docker kill dfc_ftpes-server-vsftpd1
+docker kill dfc_ftpes-server-vsftpd2
+docker kill dfc_ftpes-server-vsftpd3
+docker kill dfc_ftpes-server-vsftpd4
+docker kill dfc_cbs
+docker kill dfc_consul
+echo "Removing simulator containers"
+docker rm dfc_dr-sim
+docker rm dfc_dr-redir-sim
+docker rm dfc_mr-sim
+docker rm dfc_sftp-server0
+docker rm dfc_sftp-server1
+docker rm dfc_sftp-server2
+docker rm dfc_sftp-server3
+docker rm dfc_sftp-server4
+docker rm dfc_ftpes-server-vsftpd0
+docker rm dfc_ftpes-server-vsftpd1
+docker rm dfc_ftpes-server-vsftpd2
+docker rm dfc_ftpes-server-vsftpd3
+docker rm dfc_ftpes-server-vsftpd4
+docker rm dfc_cbs
+docker rm dfc_consul
+
+echo "done"
\ No newline at end of file
diff --git a/test/mocks/datafilecollector-testharness/simulator-group/simulators-start.sh b/test/mocks/datafilecollector-testharness/simulator-group/simulators-start.sh
index 774b753..82ad6aa 100755
--- a/test/mocks/datafilecollector-testharness/simulator-group/simulators-start.sh
+++ b/test/mocks/datafilecollector-testharness/simulator-group/simulators-start.sh
@@ -1,35 +1,115 @@
#!/bin/bash
+server_check() {
+ for i in {1..10}; do
+ res=$(curl -s -o /dev/null -w "%{http_code}" localhost:$2$3)
+ if [ $res -gt 199 ] && [ $res -lt 300 ]; then
+ echo "Simulator " $1 " on localhost:$2$3 responded ok"
+ return
+ fi
+ sleep 1
+ done
+ echo "Simulator " $1 " on localhost:$2$3 - no response"
+}
+
+ftps_server_check() {
+ for i in {1..10}; do
+ res=$(curl --silent --max-time 3 localhost:$2 2>&1 | grep vsFTPd)
+ if ! [ -z "$res" ]; then
+ echo "Simulator " $1 " on localhost:$2 responded ok"
+ return
+ fi
+ sleep 1
+ done
+ echo "Simulator " $1 " on localhost:$2 - no response"
+}
+
+sftp_server_check() {
+ for i in {1..10}; do
+ res=$(curl --silent --max-time 3 localhost:$2 2>&1 | grep OpenSSH)
+ if ! [ -z "$res" ]; then
+ echo "Simulator " $1 " on localhost:"$2" responded ok"
+ return
+ fi
+ sleep 1
+ done
+ echo "Simulator " $1 " on localhost:"$2" - no response"
+}
# Starts all simulators with the test settings
-# Intended for CSIT test. For manual start, use the docker-compose-setup.sh
+# Intended for CSIT test and auto test. For manual start, use the docker-compose-setup.sh
+
+DOCKER_SIM_NWNAME="dfcnet"
+echo "Creating docker network $DOCKER_SIM_NWNAME, if needed"
+docker network ls| grep $DOCKER_SIM_NWNAME > /dev/null || docker network create $DOCKER_SIM_NWNAME
docker-compose -f docker-compose-template.yml config > docker-compose.yml
docker-compose up -d
+declare -a SFTP_SIM
+declare -a FTPS_SIM
+
DR_SIM="$(docker ps -q --filter='name=dfc_dr-sim')"
DR_RD_SIM="$(docker ps -q --filter='name=dfc_dr-redir-sim')"
MR_SIM="$(docker ps -q --filter='name=dfc_mr-sim')"
-SFTP_SIM="$(docker ps -q --filter='name=dfc_sftp-server')"
-FTPS_SIM="$(docker ps -q --filter='name=dfc_ftpes-server-vsftpd')"
+SFTP_SIM[0]="$(docker ps -q --filter='name=dfc_sftp-server0')"
+SFTP_SIM[1]="$(docker ps -q --filter='name=dfc_sftp-server1')"
+SFTP_SIM[2]="$(docker ps -q --filter='name=dfc_sftp-server2')"
+SFTP_SIM[3]="$(docker ps -q --filter='name=dfc_sftp-server3')"
+SFTP_SIM[4]="$(docker ps -q --filter='name=dfc_sftp-server4')"
+FTPS_SIM[0]="$(docker ps -q --filter='name=dfc_ftpes-server-vsftpd0')"
+FTPS_SIM[1]="$(docker ps -q --filter='name=dfc_ftpes-server-vsftpd1')"
+FTPS_SIM[2]="$(docker ps -q --filter='name=dfc_ftpes-server-vsftpd2')"
+FTPS_SIM[3]="$(docker ps -q --filter='name=dfc_ftpes-server-vsftpd3')"
+FTPS_SIM[4]="$(docker ps -q --filter='name=dfc_ftpes-server-vsftpd4')"
+CBS_SIM="$(docker ps -q --filter='name=dfc_cbs')"
+CONSUL_SIM="$(docker ps -q --filter='name=dfc_consul')"
#Wait for initialization of docker containers for all simulators
for i in {1..10}; do
if [ $(docker inspect --format '{{ .State.Running }}' $DR_SIM) ] && \
[ $(docker inspect --format '{{ .State.Running }}' $DR_RD_SIM) ] && \
[ $(docker inspect --format '{{ .State.Running }}' $MR_SIM) ] && \
-[ $(docker inspect --format '{{ .State.Running }}' $SFTP_SIM) ] && \
-[ $(docker inspect --format '{{ .State.Running }}' $FTPS_SIM) ]
+[ $(docker inspect --format '{{ .State.Running }}' ${SFTP_SIM[0]}) ] && \
+[ $(docker inspect --format '{{ .State.Running }}' ${SFTP_SIM[1]}) ] && \
+[ $(docker inspect --format '{{ .State.Running }}' ${SFTP_SIM[2]}) ] && \
+[ $(docker inspect --format '{{ .State.Running }}' ${SFTP_SIM[3]}) ] && \
+[ $(docker inspect --format '{{ .State.Running }}' ${SFTP_SIM[4]}) ] && \
+[ $(docker inspect --format '{{ .State.Running }}' ${FTPS_SIM[0]}) ] && \
+[ $(docker inspect --format '{{ .State.Running }}' ${FTPS_SIM[1]}) ] && \
+[ $(docker inspect --format '{{ .State.Running }}' ${FTPS_SIM[2]}) ] && \
+[ $(docker inspect --format '{{ .State.Running }}' ${FTPS_SIM[3]}) ] && \
+[ $(docker inspect --format '{{ .State.Running }}' ${FTPS_SIM[4]}) ] && \
+[ $(docker inspect --format '{{ .State.Running }}' $CBS_SIM) ] && \
+[ $(docker inspect --format '{{ .State.Running }}' $CONSUL_SIM) ]
then
- echo "All simulators Running"
+ echo "All simulators Started"
break
else
echo sleep $i
sleep $i
- fi
+ fi
done
+server_check "cbs " 10000 "/healthcheck"
+server_check "consul " 8500 "/v1/catalog/service/agent"
+server_check "DR sim " 3906 "/"
+server_check "DR redir sim " 3908 "/"
+server_check "MR sim " 2222 "/"
+ftps_server_check "FTPS server 0" 1032
+ftps_server_check "FTPS server 1" 1033
+ftps_server_check "FTPS server 2" 1034
+ftps_server_check "FTPS server 3" 1035
+ftps_server_check "FTPS server 4" 1036
+sftp_server_check "SFTP server 0" 1022
+sftp_server_check "SFTP server 1" 1023
+sftp_server_check "SFTP server 2" 1024
+sftp_server_check "SFTP server 3" 1025
+sftp_server_check "SFTP server 4" 1026
+
+echo ""
+
#Populate the ftp server with files
if [ -z "$NUM_FTPFILES" ]
then
@@ -47,17 +127,35 @@
then
FTP_TYPE="ALL"
fi
+if [ -z "$FTP_FILE_PREFIXES" ]
+ then
+ FTP_FILE_PREFIXES="A"
+fi
+
+if [ -z "$NUM_FTP_SERVERS" ]
+ then
+ NUM_FTP_SERVERS=1
+fi
+
if [ $FTP_TYPE = "ALL" ] || [ $FTP_TYPE = "SFTP" ]; then
echo "Creating files for SFTP server, may take time...."
- docker cp setup-ftp-files-for-image.sh $SFTP_SIM:/tmp/
- #Double slash needed for docker on win...
- docker exec -w //home/onap/ $SFTP_SIM //tmp/setup-ftp-files-for-image.sh $NUM_FTPFILES $NUM_PNFS $FILE_SIZE #>/dev/null 2>&1
+ p=0
+ while [ $p -lt $NUM_FTP_SERVERS ]; do
+ docker cp setup-ftp-files-for-image.sh ${SFTP_SIM[$p]}:/tmp/
+ #Double slash needed for docker on win...
+ docker exec -w //home/onap/ ${SFTP_SIM[$p]} //tmp/setup-ftp-files-for-image.sh $NUM_FTPFILES $NUM_PNFS $FILE_SIZE $FTP_FILE_PREFIXES $NUM_FTP_SERVERS $p #>/dev/null 2>&1
+ let p=p+1
+ done
fi
if [ $FTP_TYPE = "ALL" ] || [ $FTP_TYPE = "FTPS" ]; then
echo "Creating files for FTPS server, may take time...."
- docker cp setup-ftp-files-for-image.sh $FTPS_SIM:/tmp/setup-ftp-files-for-image.sh
- #Double slash needed for docker on win...
- docker exec -w //srv $FTPS_SIM //tmp/setup-ftp-files-for-image.sh $NUM_FTPFILES $NUM_PNFS $FILE_SIZE #>/dev/null 2>&1
+ p=0
+ while [ $p -lt $NUM_FTP_SERVERS ]; do
+ docker cp setup-ftp-files-for-image.sh ${FTPS_SIM[$p]}:/tmp/setup-ftp-files-for-image.sh
+ #Double slash needed for docker on win...
+ docker exec -w //srv ${FTPS_SIM[$p]} //tmp/setup-ftp-files-for-image.sh $NUM_FTPFILES $NUM_PNFS $FILE_SIZE $FTP_FILE_PREFIXES $NUM_FTP_SERVERS $p #>/dev/null 2>&1
+ let p=p+1
+ done
fi
echo "Done: All simulators started and configured"