Welcome to FIWARE Monitoring¶

Components¶

Monitoring framework

It is up to the infrastructure owner which tool (like Nagios, Zabbix, openNMS, perfSONAR, etc.) is installed for this purpose.

Collector

Framework-specific component to forward monitoring data being gathered to the adaptation layer (i.e. NGSI Adapter). Monitoring GE provides a Nagios loadable module NGSI Event Broker as collector for such monitoring framework.

Adaptation layer

NGSI Adapter serves as generic adapter to transform monitoring data from probes to NGSI context attributes.

Requirements¶

• System resources: see these recommendations.
• Operating systems: CentOS (RedHat) and Ubuntu (Debian), being CentOS 6.3 the reference operating system.
• RPM/DEB dependencies: some required packages may not be present in official repositories, or their versions are too old (for example, nodejs). In any case, checking for such dependencies and configuration of alternative sources is automatically managed by the package installation scripts when using the proper tool (yum in CentOS or apt-get/gdebi in Ubuntu).

Installation¶

http://repositories.testbed.fiware.org/repo/rpm/x86_64

http://repositories.testbed.fiware.org/repo/deb

$ sudo yum install fiware-monitoring-ngsi-adapter

$ sudo apt-get install fiware-monitoring-ngsi-adapter

$ sudo rpm -i fiware-monitoring-ngsi-adapter-X.Y.Z-1.noarch.rpm
$ sudo rpm -i fiware-monitoring-ngsi-event-broker-X.Y.Z-1.x86_64.rpm

$ sudo dpkg -i fiware-monitoring-ngsi-adapter_X.Y.Z_all.deb
$ sudo dpkg -i fiware-monitoring-ngsi-event-broker_X.Y.Z_amd64.deb

Upgrading from a previous version¶

Unless explicitly stated, no migration steps are required to upgrade to a newer version of the Monitoring components:

• When using the package repositories, just follow the same directions described in the Installation section (the install subcommand also performs upgrades).
• When upgrading from downloaded package files, use rpm -U in CentOS, or use same dpkg -i command in Ubuntu.

Configuration file¶

The configuration used by the adapter service is optionally read from the file /etc/sysconfig/ngsi_adapter (in CentOS) or /etc/default/ngsi_adapter (in Ubuntu):

# ADAPTER_LOGFILE - Logging file
ADAPTER_LOGFILE=/var/log/ngsi_adapter/ngsi_adapter.log

# ADAPTER_LOGLEVEL - Logging level
ADAPTER_LOGLEVEL=INFO

# ADAPTER_LISTEN_HOST - The host where NGSI Adapter listens to requests
ADAPTER_LISTEN_HOST=0.0.0.0

# ADAPTER_LISTEN_PORT - The port where NGSI Adapter listens to requests
ADAPTER_LISTEN_PORT=1337

# ADAPTER_UDP_ENDPOINTS - UDP listen endpoints (host:port:parser,...)

# ADAPTER_PARSERS_PATH - Path with directories to look for parsers
ADAPTER_PARSERS_PATH=lib/parsers/nagios

# ADAPTER_BROKER_URL - The endpoint where Context Broker is listening
ADAPTER_BROKER_URL=http://127.0.0.1:1026/

# ADAPTER_MAX_REQUESTS - Maximum number of simultaneous requests
ADAPTER_MAX_REQUESTS=5

# ADAPTER_RETRIES - Maximum number of retries invoking Context Broker
ADAPTER_RETRIES=2

Most of these attributes map to options of the command line interface as follows:

• ADAPTER_LOGLEVEL maps to -l or --logLevel option
• ADAPTER_LISTEN_HOST maps to -H or --listenHost option
• ADAPTER_LISTEN_PORT maps to -p or --listenPort option
• ADAPTER_UDP_ENDPOINTS maps to -u or --udpEndpoints option
• ADAPTER_PARSERS_PATH maps to -P or --parsersPath option
• ADAPTER_BROKER_URL maps to -b or --brokerUrl option
• ADAPTER_MAX_REQUESTS maps to -m or --maxRequests option
• ADAPTER_RETRIES maps to -r or --retries option

Default values are found in /opt/fiware/ngsi_adapter/lib/common.js.

Checking status¶

In order to check the status of the adapter service, use the following command (no special privileges required):

$ service ngsi_adapter status

API Reference Documentation¶

• FIWARE Monitoring v1 (Apiary)

End-to-end tests¶

Please refer to the Installation and administration guide for details.

Unit tests¶

The test target is used for running the unit tests in both components of Monitoring GE:

$ cd ngsi_adapter
$ grunt test

$ cd ngsi_event_broker
$ make test  # synonym of standard 'check' target

Please have a look at the section building from source code in order to get more information about how to prepare the environment to run the unit tests.

Acceptance tests¶

In the following documents you will find a business readable description of the features provided by the components of the Monitoring GE, as well as automated tests for them:

• NGSI Adapter acceptance tests

Background and Detail¶

This User and Programmers Guide relates to the Scalability Manager GE which is part of the Cloud Hosting Chapter. Please find more information about this Generic Enabler in the following Open Specification.

NGSI Adapter API (HTTP)¶

Probe raw data should be sent as body of a POST request to the adapter, identifying the source entity being monitored in the query parameters. For example, given the following scenario:

then requests would look like:

HTTP POST http://adapterhost:1337/check_load?id=178.23.5.23&type=host
Content-Type: text/plain
OK - load average: 0.36, 0.25, 0.24|load1=0.360;1.000;1.000;0;
load5=0.250;5.000;5.000;0; load15=0.240;15.000;15.000;0;

Please take into account that NGSI standard identify entities (in this case, the resources being monitored) using a pair <entityId,entityType>. This identification of the monitored resource has to be provided as the query parameters id and type, respectively. The probe name included in the URL lets NGSI Adapter know the originating monitoring probe, therefore selecting the proper parser for it. This API is fully described in Apiary.

Monitoring framework is expected to schedule the execution of probes and send the raw data been gathered to the NGSI Adapter. Depending on the tool that has been chosen, this would require the development of a custom component (a kind of monitoring collector) to automatically forward such data to the adaptation layer.

NGSI Adapter API (UDP)¶

In case UDP endpoints are defined (specifying the target parser to be loaded), probe raw data should be sent as UDP request to the adapter. Such message is expected to include both the id and the type of the NGSI Entity whose data is about to be parsed.

NGSI Adapter parsers¶

NGSI Adapter processes requests asynchronously, trying to load a valid parser named after the originating probe, located at any of the directories specified (see Installation and Administration Guide). If probe is unknown (parser not found), HTTP response status will be 404; otherwise, response status will be 200, parser will be dynamically loaded, and then its parseRequest() and getContextAttrs() methods will be called. The attribute list returned by the latter will be used to invoke Context Broker.

Custom parsers for new probes may be easily added to NGSI Adapter, just extending a base abstract object and implementing the aforementioned methods. For example, suppose we want to support a new “myProbe” whose data is a comma-separated list of values of two attributes myAttr0 and myAttr1:

/**
 * module "myProbe" at any directory included in ADAPTER_PARSERS_PATH
 */

// @augments base parser (must redefine parseRequest and getContextAttrs)
var myParser = Object.create(null);

// @param Domain object including context, timestamp, id, type & body
myParser.parseRequest = function (reqDomain) {
    var reqDataContent = this.doSomeParsing(reqDomain.body);
    return { data: reqDataContent };
};

// @param EntityData object including data attribute
myParser.getContextAttrs = function (entityData) {
    var items = this.doMoreParsing(entityData.data);
    return { myAttr0: items[0], myAttr1: items[1] };
};

exports.parser = myParser;

Custom parsers for UDP request must also set the attributes entityId and entityType of the input object reqDomain on return, given that such information is part of the UDP message itself being parsed:

// @param Domain object
myParser.parseRequest = function (reqDomain) {
    var identification = this.doSomeParsing(reqDomain.body),
        reqDataContent = this.doMoreParsing(reqDomain.body);
    reqDomain.entityId   = identification['id'];
    reqDomain.entityType = identification['type'];
    return { data: reqDataContent };
};

Context Broker API¶

Please refer to Context Broker documentation. This will give us access to the last updates of monitoring data available, but not to historical data.

Monitoring API¶

Retrieval of historical data stored at a distributed filesystem (e.g. Hadoop) is handled by the Query Manager component, whose API is described in this preliminary specification.

Installation of probes¶

Monitoring GE is agnostic to the framework used to gather monitoring data. It just assumes there are several probes collecting such data, which somehow will be forwarded to the adaptation layer (NGSI Adapter).

It is up to the infrastructure owner which tool (like Nagios, Zabbix, openNMS, perfSONAR, etc.) is installed for this purpose.

Installation of collector¶

Probes must “publish” their data to NGSI Adapter. Depending on the exact monitoring tool installed, a kind of collector has to be deployed in order to send data to the adapter:

• NGSI Event Broker is an example specific for Nagios, implemented as a loadable module. Description and installation details can be found here.

Installation of NGSI Adapter¶

Installation of Context Broker¶

Please refer to Orion documentation.

Installation of the connector¶

This component subscribes to changes at Context Broker and writes data into a distributed filesystem storage (usually HDFS from Hadoop). Historically the ngsi2cosmos connector implementation has been used (installation details here), although from March 2014 this component is deprecated and a brand new Cygnus implementation (installation details here) is available.

Running NGSI Adapter¶

Once installed, there are two ways of running NGSI Adapter: manually from the command line or as a system service. It is not recommended to mix both ways (e.g. start it manually but using the service scripts to stop it).

$ adapter

End to end testing¶

Use the commands of the monitoring framework being used (for example, Nagios) to reschedule some probe execution and force the generation of new monitoring data:

• Check the logs of the framework (i.e. /var/log/nagios/nagios.log) for a new probe execution detected by the collector:

  $ cat /var/log/nagios/nagios.log
  [1439283831] lvl=INFO | trans=rdPmJ/uHE62a |
                 comp=fiware-monitoring-ngsi-event-broker | op=NGSIAdapter |
                 msg=Request sent to http://host:1337/check_xxx?id=xxx&type=host
  

• Check NGSI Adapter logs for incoming requests with raw data, and for the corresponding updateContext() request to Context Broker:

  $ cat /var/log/ngsi_adapter/ngsi_adapter.log
  time=... | lvl=INFO | trans=rdPmJ/uHE62a | op=POST |
                 msg=Request on resource /check_xxx with params id=xxx&type=xxx
  
  time=... | lvl=INFO | trans=rdPmJ/uHE62a | op=POST | msg=Response status 200 OK
  time=... | lvl=INFO | trans=rdPmJ/uHE62a | op=UpdateContext |
                 msg=Request to ContextBroker at http://host:1026/...
  

• Finally, query Context Broker API to check whether entity attributes have been updated according to the new monitoring data (see details here)

List of Running Processes¶

A node process running the “adapter” server should be up and running, e.g.:

$ ps -C node -f | grep adapter
fiware   21930     1  0 Mar28 ?        00:06:06 node /opt/fiware/ngsi_adapter/adapter

Alternatively, we can check if service is running, e.g.:

$ service ngsi_adapter status
* ngsi_adapter is running

Network interfaces Up & Open¶

NGSI Adapter uses TCP 1337 as default port, although it can be changed using the --listenPort command line option.

Additionally, a list of UDP listen ports may be specified by --udpEndpoints command line option.

Databases¶

This component does not persist any data, and no database engine is needed.

Resource availability¶

Although we haven’t done yet a precise profiling on NGSI Adapter, tests done in our development and testing environment show that a host with 2 CPU cores and 4 GB RAM is fine to run server.

Remote service access¶

• Probes at monitored hosts should have access to NGSI Adapter listen port (TCP 1337, by default)
• NGSI Adapter should have access to Context Broker listen port (TCP 1026, by default)
• Connector should have access to Context Broker listen port in order to subscribe to context changes
• Context Broker should have access to Connector callback port to notify changes

Resource consumption¶

No issues related to resources consumption have been detected neither with the NGSI Adapter server nor with the NGSI Event Broker loaded as a “pluggable” module on Nagios startup.

I/O flows¶

Figure at installation section shows the I/O flows among the different monitoring components:

• Probes send requests to NGSI Adapter with raw monitoring data, by means of a custom collector component (for example, NGSI Event Broker)
• NGSI Adapter sends request to Context Broker in terms of context updates of the monitored resources
• Context Broker notifies Connector with every context change
• Connector writes changes to storage