6.6. Event Notification

RackHD supports event notification via both web hook and AMQP.

A web hook allows applications to subscribe certain RackHD published events by configured URL, when one of the subscribed events is triggered, RackHD will send a POST request with event payload to configured URL.

RackHD also publishes defined events over AMQP, so subscribers to RackHD’s instance of AMQP don’t need to register a webhook URL to get events. The AMQP events can be prolific, so we recommend that consumers filter events as they are received to what is desired.

6.6.1. Events Payloads

All published external events’ payload formats are common, the event attributes are as below:

Attribute Type Description
version String Event payload format version.
type String It could be one of the values: heartbeat, node, polleralert, graph.
action String a verb or a composition of component and verb which indicates what happened, it’s associated with the type attribute.
severity String Event severity, it could be one of the values: critical, warning, information.
typeId String It’s associated with the type attribute. It could be graph ‘Id’ for graph type, poller ‘Id’ for polleralert type, <fqdn>.<service name> for heartbeat event, node ‘Id’ for node type. Please see table for more details .
createdAt String The time event happened.
nodeId String The node Id, it’s null for ‘heartbeat’ event.
data Object Detail information are included in this attribute.

The table of type, typeId, action and severity for all external events

Example of heartbeat event payload:

{
    "version": "1.0",
    "type": "heartbeat",
    "action": "updated",
    "typeId": "kickseed.example.com.on-taskgraph",
    "severity": "information",
    "createdAt": "2016-07-13T14:23:45.627Z",
    "nodeId": "null",
    "data": {
        "name": "on-taskgraph",
        "title": "node",
        "pid": 6086,
        "uid": 0,
        "platform": "linux",
        "release": {
            "name": "node",
            "lts": "Argon",
            "sourceUrl": "https://nodejs.org/download/release/v4.7.2/node-v4.7.2.tar.gz",
            "headersUrl": "https://nodejs.org/download/release/v4.7.2/node-v4.7.2-headers.tar.gz"
        },
        "versions": {
            "http_parser": "2.7.0",
            "node": "4.7.2",
            "v8": "4.5.103.43",
            "uv": "1.9.1",
            "zlib": "1.2.8",
            "ares": "1.10.1-DEV",
            "icu": "56.1",
            "modules": "46",
            "openssl": "1.0.2j"
        },
        "memoryUsage": {
            "rss": 116531200,
            "heapTotal": 84715104,
            "heapUsed": 81638904
        },
        "currentTime": "2017-01-24T07:18:49.236Z",
        "nextUpdate": "2017-01-24T07:18:59.236Z",
        "lastUpdate": "2017-01-24T07:18:39.236Z",
        "cpuUsage": "NA"
    }
}

Example of node discovered event payload:

{
    "type": "node",
    "action": "discovered",
    "typeId": "58aa8e54ef2b49ed6a6cdd4c",
    "nodeId": "58aa8e54ef2b49ed6a6cdd4c",
    "severity": "information",
    "data": {
        "ipMacAddresses": [
            {
                "ipAddress": "172.31.128.2",
                "macAddress": "2c:60:0c:ad:d5:ba"
            },
            {
                "macAddress": "90:e2:ba:91:1b:e4"
            },
            {
                "macAddress": "90:e2:ba:91:1b:e5"
            },
            {
                "macAddress": "2c:60:0c:c0:a8:ce"
            }
        ],
        "nodeId": "58aa8e54ef2b49ed6a6cdd4c",
        "nodeType": "compute"
    },
    "version": "1.0",
    "createdAt": "2017-02-20T06:37:23.775Z"
}

6.6.2. Events via AMQP

6.6.2.1. AMQP Exchange and Routing Key

The change of resources managed by RackHD could be retrieved from AMQP messages.

  • Exchange: on.events
  • Routing Key <type>.<action>.<severity>.<typeId>.<nodeId>

ALl the fields in routing key exists in the common event payloads event_payload.

Examples of routing key:

Heartbeat event routing key of on-tftp service:

heartbeat.updated.information.kickseed.example.com.on-tftp

Polleralert sel event routing key:

polleralert.sel.updated.critical.44b15c51450be454180fabc.57b15c51450be454180fa460

Node discovered event routing key:

node.discovered.information.57b15c51450be454180fa460.57b15c51450be454180fa460

Graph event routing key:

graph.started.information.35b15c51450be454180fabd.57b15c51450be454180fa460

6.6.2.2. AMQP Routing Key Filter

All the events could be filtered by routing keys, for example:

All services’ heartbeat events:

$ sudo node sniff.js "on.events" "heartbeat.#"

All nodes’ discovered events:

$ sudo node sniff.js "on.events" "#.discovered.#"

‘sniff.js’ is a tool located at https://github.com/RackHD/on-tools/blob/master/dev_tools/README.md

6.6.3. Events via Hook

6.6.3.1. Register Web Hooks

The web hooks used for subscribing event notification could be registered by POST <server>/api/current/hooks API as below

curl -H "Content-Type: application/json" -X POST -d @payload.json <server>api/current/hooks

The payload.json attributes in the example above are as below:

Attribute Type Flags Description
url String required The hook url that events are notified to. Both http and https urls are supported. url must be unique.
name String optional Any name user specified for the hook.
filters Array optional An array of conditions that decides which events should be notified to hook url.

When a hook is registered and eligible events happened, RackHD will send a POST request to the hook url. POST request’s Content-Type will be application/json, and the request body be the event payload.

An example of payload.json with minimal attributes:

{
    "url": "http://www.abc.com/def"
}

When multiple hooks are registered, a single event can be sent to multiple hook urls if it meets hooks’ filtering conditions.

6.6.3.2. Event Filter Rules

The conditions of which events should be notified could be specified in the filters attribute in the hook_payload, when filters attribute is not specified, or it’s empty, all the events will be notified to the hook url.

The filters attribute is an array, so multiple filters could be specified. The event will be sent as long as any filter condition is satisfied, even if the conditions may have overlaps.

The filter attributes are type, typeId, action, severity and nodeId listed in event_payload. Filtering by data is not supported currently. Filtering expression of hook filters is based on javascript regular expression, below table describes some base operations for hook filters:

Description Example Eligible Events
Attribute equals some value {“action”: “^discovered$”} Events with action equals discovered
Attribute can be any of specified value. {“action”: “discovered|updated”} Events with action equals either discovered or updated
Attribute can not be any of specified value. {“action”: “[^(discovered|updated)]”} Events with action equals neither discovered nor updated
Multiple attributes must meet specified values. {“action”: “[^(discovered|updated)]”, “type”: “node”} Events with type equals node while action equals neither discovered nor updated

An example of multiple filters:

{
    "name": "event sets",
    "url": "http://www.abc.com/def",
    "filters": [
        {
            "type": "node",
            "nodeId": "57b15c51450be454180fa460"
        },
        {
            "type": "node",
            "action": "discovered|updated",
        }
    ]
}

6.6.3.3. Web Hook APIs

Create a new hook

POST /api/2.0/hooks
{
    "url": "http://www.abc.com/def"
}

Delete an existing hook

DELETE /api/2.0/hooks/:id

Get a list of hooks

GET /api/2.0/hooks

Get details of a single hook

GET /api/2.0/hooks/:id

Update an existing hook

PATCH /api/2.0/hooks/:id
{
    "name": "New Hook"
}

6.6.4. Redfish Alert Notification

6.6.4.1. Description

RackHD is enabled to receive redfish based notifications. It is possible to configure a redfish endpoint to send alerts to RackHD. When RackHD receives an alert, it determines which node issued the alert and then it adds some additional context such as nodeId, service tag, etc. Lastly, RackHD publishes the alert to AMQP and Web Hook.

6.6.4.2. Configuring the Redfish endpoint

If the endpoint is redfish enabled and supports the Resfish EventService, it is possible to configure the endpoint to send the alerts to RackHD. Please note that the “Destination” property in the example below should be a reference to RackHD.

POST /redfish/v1/EventService/Subscriptions
    {
            "Context": "context string",
            "Description": "Event Subscription Details",
            "Destination": "https://10.240.19.226:8443/api/2.0/notification/alerts",
            "EventTypes": [
            "ResourceAdded",
            "StatusChange",
                "Alert"
            ],
            "Id": "id",
            "Name": "name",
            "Protocol": "Redfish"
    }

6.6.4.3. Alert message

In addition to the redfish alert message, RackHD adds the following properties: “sourceIpAddress” (of the BMC), “nodeId”,”macAddress” (of the BMC), “ChassisName”, “ServiceTag”, “SN”.

{
        "type": "node",
        "action": "alerts",
        "data": {
                "Context": "context string",
                "EventId": "8689",
                "EventTimestamp": "2017-04-03T10:07:32-0500",
                "EventType": "Alert",
                "MemberId": "7e675c8e-127a-11e7-9fc8-64006ac35232",
                "Message": "The coin cell battery in CMC 1 is not working.",
                "MessageArgs": ["1"],
                "MessageArgs@odata.count": 1,
                "MessageId": "CMC8572",
                "Severity": "Critical",
                "sourceIpAddress": "10.240.19.130",
                "nodeId": "58d94cec316779d4126be134",
                "macAddress": "64:00:6a:c3:52:32",
                "ChassisName": "PowerEdge R630",
                "ServiceTag": "4666482",
                "SN": "CN747515A80855"
        },
        "severity": "critical",
        "typeId": "58d94cec316779d4126be134",
        "version": "1.0",
        "createdAt": "2017-04-03T14:11:46.245Z"
}

6.6.4.4. AMQP

The messages are pulished to:

  • Exchange: on.events
  • Routing Key: node.alerts.<severity>.<typeId>.<nodeId>