commit | dc936d27d761bde31ac5916a84efa2f48ec32b83 | [log] [tgz] |
---|---|---|
author | Filip Krzywka <filip.krzywka@nokia.com> | Fri May 10 11:53:21 2019 +0200 |
committer | Filip Krzywka <filip.krzywka@nokia.com> | Fri May 10 11:53:51 2019 +0200 |
tree | 0c52440545105d137d13c238cb54a2e0c364d167 | |
parent | e3b124ab11ac8c837ff692539696d1e43fcd8247 [diff] |
Enable Kafka consumer offset committing It appears that reactor-kafka is setting auto.commit property to false, which makes our CSITs fail nondeterministically due to automatic reset of consumer offset. By acknowledging manually, we will mark every message for committing, which will be performed according to ConsumerConfiguration. This way, Kafka broker should persist consumer offset. Change-Id: I0c5156ff8df9bb3341e733e50a3c6866fdd94976 Issue-ID: DCAEGEN2-1495 Signed-off-by: Filip Krzywka <filip.krzywka@nokia.com>
ONAP component for collecting high volume data, eg, RTPM (Real Time Performance Management).
##General information
##Background VES-HV collector has been proposed, based on a need to process high-volumes of data generated frequently by a large number of NFs. The driving use-case is the 5G RAN, where it is expected that up to 10k NF instances report the data, per DCAE platform deployment. The network traffic generated in simulations - based on 4G BTS Real-Time PM data has shown, that GPB serialization is 2-3 times more effective, than JSON serialization utilized in VES collector. Results have been published within ONAP presentation in Casablanca Release Developer Forum: Google Protocol Buffers versus JSON - 5G RAN use-case - comparison
The goal of the collector is to support high volume data. It uses plain TCP connections tunneled in SSL/TLS. Connections are stream-based (as opposed to request-based) and long running. Payload is binary-encoded (currently we are using Google Protocol Buffers). HV-VES uses direct connection to DMaaP's Kafka. All these decisions were made in order to support high-volume data with minimal latency.
For more details on the rationale, please read a high-level feature description.
##Description Compatibility aspects (VES-JSON)
Extendability
VES-HV was designed to allow for extendability - by adding new domain-specific PROTO files.
The PROTO file, which contains the VES CommonHeader, comes with a binary-type Payload parameter, where domain-specific data shall be placed. Domain-specific data are encoded as well with GPB, and they do require a domain-specific PROTO file to decode the data. This domain-specific PROTO needs to be shared with analytics applications - VES-HV is not analyzing domain-specific data. In order to support the RT-PM use-case, VES-HV includes a "perf3gpp" domain PROTO file, as within this domain, the high volume data is expected to be reported to VES-HV collector. Still, there are no limitations to define additional domains, based on existing VES domains (like Fault, Heartbeat) or completely new domains. New domains can be added "when needed".
In case of new domains, it is necessary to extend the Common Header PROTO "Domain" enumeration with new values covering this new domain(s). GPB PROTO files are backwards compatible, and such a new domain could be added without affecting existing systems.
Analytics applications will have to be as well equipped with this new domain-specific PROTO file. Currently, these additional, domain specific proto files could be simply added to respective repos of VES-HV collector.
##Implementation details Technology stack
Rules