RackHD™¶

Applications¶

Libraries¶

Supplemental Code¶

Documentation¶

Repositories Status¶

The following badges in the tables may take a while to load.

Repository	Travis-Ci Build	Code Climate	Code Coverage
on-core
on-dhcp-proxy
on-http
on-imagebuilder		N/A	N/A
on-statsd
on-syslog
on-taskgraph
on-tasks
on-tftp
on-web-ui			N/A
on-wss			N/A

Guidelines for merging pull requests¶

For code changes, we currently use a guideline of lazy consensus with two positive reviews with at least one of those reviews being one of the core maintainers and no negative votes. And of course, the gates for the pull requests must pass as well (unit tests, etc).

If you put a review up, please be explicit with a vote (+1, -1, or +/-0) so we can distinguish questions asking for information or background from reviews implying that the relevant change should not be merged. Likewise if you put up a change for review as a pull request, a -1 review comment isn’t a reflection on you as a person, instead is a request to make a modification before that pull request should be merged.

For those with commit privileges

See https://github.com/RackHD/RackHD/wiki/Merge-Guidelines for more informal guidelines and rules of thumb to follow when making merge decisions.

Getting commit privileges¶

The core committer team will grant contributor rights to the RackHD project using a lazy consensus mechanism. Any of the maintainers/core contributors can nominate someone to have those privileges, and with two +1 votes and no negative votes, the team will grant commit privileges.

The core team will also be responsible for removing commit privileges when appropriate - for example for malicious merge behavior or just inactivity over an extended period of time.

Quality gates for the pull requests¶

There are three quality gates to ensure the pull requests quality, Hound for code style check, Travis CI for unit-test and coveralls, Jenkins for the combination test including unit-test and smoke test. When a pull request is created, all tests will run automatically, and the test results can be found in the merge status field of each pull request page. Running unit/functional tests locally prior to creating a pull request is strongly encouraged. This would hopefully minimize the amount errors seen during PR submission and lessen a dependency on Travis/Jenkins to test code before it’s really ready to be submitted.

Hound

Hound works with jshint and comments on style violations in pull requests. Configuration files .hound.yml and .jshintrc have been created in each repository, so before creating a pull request, you can check code style locally with jshint to find out style violations beforehand.

Travis CI

Travis CI runs the unit tests, and then does some potentially ancillary actions. The build specifics are detailed in the .travis.yml file within each repository. For finding out basic errors before creating a pull request, you can run unit test locally using npm test within each repository.

Concourse

RackHD uses Concourse CI to monitor and perform quality gate tests on all pull requests prior to merge. The gates include running all the unit tests, running all dependent project unit tests with the code proposed from the pull request, running an integration “smoke test” to verify basic end to end functionality and commenting on the details of test case failure. Concourse can also take instructions from pull request comments or description in order to handle more complex test scenarios. Instructions can be written in the pull request description or comments.

All pull requests will need to be labeled with the “run-test” label before the quality gate tests will run. This label needs to be set by a RackHD Commit.

The following table show all the Jenkins Instructions and usage:

depends on: pr1_url
depends on: pr2_url
...

Workflows¶

We use the following conventions when creating workflow-related JSON documents:

Tasks

For task definitions, the only convention is for values in the “injectableName” field. We tend to prefix all names with “Task.” and then add some categorization to classify what functionality the task adds.

Examples:

Task.Os.Install.CentOS
Task.Os.Install.Ubuntu
Task.Obm.Node.PowerOff
Task.Obm.Node.PowerOn

Graphs

For graph definitions, conventions are pretty much the same as tasks, except “injectableName” is prefixed by “Graph.”.

Examples:

Graph.Arista.Zerotouch.vEOS
Graph.Arista.Zerotouch.EOS

Microkernel docker image¶

Image Names

We tend to prefix docker images with micro_ along with some information about which RancherOS the docker image was built off and information about what is contained within the docker image. Images are suffixed with docker.tar.xz because they are xzed tar archives contain docker image.

Examples:

micro_1.2.0_flashupdt.docker.tar.xz
micro_1.2.0_brocade.docker.tar.xz
micro_1.2.0_all_binaries.docker.tar.xz

Image Files

When adding scripts and binaries to docker image, we typically put them in /opt within subdirectories based on vendor.

Examples:

/opt/MegaRAID/MegaCli/MegaCli64
/opt/MegaRAID/StorCli/storcli64
/opt/mpt/mpt3fusion/sas3flash

If you want to add binaries or scripts and reference them by name rather than their absolute paths, then add them to /usr/local/bin or any other directory in the default PATH for bash.

File Paths

Our HTTP server will serve docker images from /opt/monorail/static/http. It is recommended that you create subdirectories within this directory for further organization.

Examples:

/opt/monorail/static/http/teamA/intel_flashing/micro_1.2.0_flashupdt.docker.tar.xz
/opt/monorail/static/http/teamA/generic/micro_1.2.0_all_binaries.docker.tar.xz

These file paths can then be referenced in workflows starting from the base path of /opt/monorail/static/http, so the above paths are referenced for download as:

teamA/intel_flashing/micro_1.2.0_flashupdt.docker.tar.xz
teamA/generic/micro_1.2.0_all_binaries.docker.tar.xz

Referencing API Versions in URIs¶

Use the following convention when referencing API version:

/api/current/...
/api/1.1/...
/api/1.2/...

The second /[...]/ block in the URI is the version number. The “current” or “latest” placeholder points to the latest version of the API in the system.

Multiple API versions can be added in parallel. Use N, N-1, N-2, etc. as the naming convention.

All API versioning information should be conveyed in HTTP headers.

Versioning Resources¶

A translation and validation chain is used to support versioned “types” for URI resources from the RackHD system. The chain flow is:

BUSINESS OBJECT — TRANSLATE — VALIDATE

Data objects should be versioned in line with the API version.

API Version Guidelines¶

Use the following guide lines when determining if a new API version is needed.

The following changes require a new API version:

• changing the semantic meaning of a URI route
• removing a URI route

The following changes do not require a new API version:

• adding an entirely new URI route
• changing the query parameters (pagination, filtering, etc.) accepted by the URI route
• changing the return values on error conditions
• changing the data structure for a resource at a given URI

Configuration¶

RPC channel for making dynamic system configuration changes

Routing keys:

methods.set
methods.get

Events¶

One to many broadcast of events applicable to workflows and reactions (where poller/telemetry events will be placed in the future as well)

Routing keys:

tftp.success.[nodeid]
tftp.failure.[nodeid]
http.response.[nodeid]
dhcp.bind.success.[nodeid]
task.finished.[taskid]
graph.started.[graphid]
graph.finished.[graphid]
sku.assigned.[nodeid]

HTTP¶

Routing keys:

http.response

(uncertain - duplicate of http.response.[nodeid]?)

DHCP¶

RPC channel for interrogating the DHCP service

Routing keys:

methods.lookupIpLease
methods.ipInRange
methods.peekLeaseTable
methods.removeLease
methods.removeLeaseByIp
methods.pinMac
methods.unpinMac
methods.pinIp
methods.unpinIp

TFTP¶

(nothing defined)

Logging¶

Routing keys:

emerg
alert
error
warning
notice
info
debug
silly

task-graph-runner¶

RPC mechanism for communicating with process running workflows

Routing keys:

methods.getTaskGraphLibrary
methods.getTaskLibrary
methods.getActiveTaskGraph
methods.getActiveTaskGraphs
methods.defineTaskGraph
methods.defineTask
methods.runTaskGraph
methods.cancelTaskGraph
methods.pauseTaskGraph
methods.resumeTaskGraph
methods.getTaskGraphProperties

Scheduler¶

RPC mechanism for scheduling tasks within a workflow to run

schedule

Task¶

RPC mechanism for tasks to interrogate or interact with workflows (task-graphs)

run.[taskid]
cancel.[taskid]
methods.requestProfile.[id] (right now, nodeId)
methods.requestProperties.[id] (right now, nodeId)
methods.requestCommands.[id] (right now, nodeId)
methods.respondCommands.[id] (right now, nodeId)
methods.getBootProfile.[nodeid]
methods.activeTaskExists.[nodeId]
methods.requestPollerCache
ipmi.command.[command].[graphid] (right now, command is 'power', 'sel' or 'sdr')
ipmi.command.[command].result.[graphid] (right now, command is 'power', 'sel' or 'sdr')
run.snmp.command.[graphid]
snmp.command.result.[graphid]
poller.alert.[graphid]

Publish (Exchange, Topic, Data) -> Promise (Success)¶

Publish provides the mechanism to send data to a particular RabbitMQ exchange & topic.

Subscribe (Exchange, Topic, Callback) -> Promise (Subscription)¶

Subscribe provides the mechanism to listen for publishes or requests which are provided through the callback argument. The subscribe callback receives data in the form of the following:

function (data, message) {
    /*
     *  data - The published message data.
     *  message - A Message object with additional data and features.
     */
}

To respond to a message we support the Promise deferred syntax.

Success

message.resolve({ hello: 'world' });

Failure

message.reject(new Error('Some Error'));

Request (Exchange, Topic, Data) -> Promise (Response)¶

Request is a wrapper around the Publish/Subscribe mechanism which will first create a reply queue for a response and then publish the data to the requested exchange & topic. It’s assumed that a Subscriber using the Subscribe API will respond to the message or a timeout will occur. The reply queue is automatically generated and disposed of at the end of the request so no subscriptions need to be managed by the consumer.

Object Marshaling¶

While plain JavaScript objects can be sent over the messenger it also supports marshaling of Serializable types in On-Core. Objects which implement the Serializable interface can be marshaled over AMQP by using a constructor initialization convention and by registering their type with the messenger. When sending a Serializable object over AMQP the messenger uses the registered type to decorate the AMQP message in a way in which a receiver can create a new copy of the object using it’s typed constructor. Subscribers who receive constructed types will have access to them directly through their data value in the subscriber callback.

Object Validation¶

On publish and on subscription callback the messenger will also validate Serializable objects using the Validatable base class. Validation is provided via JSON Schemas which are attached to the sub-classed Validatable objects. If an object to be marshaled is Validatable the messenger will validate the object prior to publish or subscribe callback. Future versions of the messenger will support subscription and request type definitions which will allow consumers to identify what types of objects they expect to be notified about which will give the messenger an additional means of ensuring communications are handled correctly. Some example schemas are listed below: MAC Address

{
    id: 'MacAddress',
    type: 'object',
    properties: {
        value: {
            type: 'string',
            pattern: '^([0-9a-fA-F][0-9a-fA-F]:){5}([0-9a-fA-F][0-9a-fA-F])$'
        }
    },
    required: [ 'value' ]
}

IP Address

{
    id: 'IpAddress',
    type: 'object',
    properties: {
        value: {
            type: 'string',
            format: 'ipv4'
        }
    },
    required: [ 'value' ]
}

Lookup Model (via On-Http)

{
    id: 'Serializables.V1.Lookup',
    type: 'object',
    properties: {
        node: {
            type: 'string'
        },
        ipAddress: {
            type: 'string',
            format: 'ipv4'
        },
        macAddress: {
            type: 'string',
            pattern: '^([0-9A-Fa-f]{2}[:-]){5}([0-9A-Fa-f]{2})$'
        }
    },
    required: [ 'macAddress' ]
}

Additional Information¶

With the primary goal of the messenger being to simplify usage patterns for the consumer not all of the features have been highlighted. Below is a quick recap of the high level features.

• Publish, Subscribe, and Request/Response Patterns.
• Optional Object Marshaling.
• Optional Object Validation via JSON Schema.
• Publish & Subscribe use their own connections to improve latency in request/response patterns.
• Automatic creation of exchanges on startup.
• Automatic subscription management for Request/Response patterns.
• Automatic Request correlation and context marshaling.

Log Levels¶

We have a common set of logging levels within RackHD, used across the projects and applications. The levels are defined in the on-core library

The conventions for using the levels are:

critical

Used for logging terminal failures that are crashing the system, for information to support post-failure debugging. Errors logged as critical are expected to be terminal and will likely result in the application crashing or failing to start.

Errors logged at a critical level should be actionable in that the tracebacks or logged errors should allow resolution of the error with a code or configuration update. These errors are generally considered failures of the program to anticipate corner conditions or failure modes.

error

Logging errors that may (or will) result in the application behaving in an unexpected fashion. Assertion/precondition errors are appropriate here, as well as any error that would generate an “unknown” error and be exposed via a 500 response (i.e. an undefined error) in an HTTP response code. The results of these errors are not expected to be terminal to the operation of the application.

Errors logged at an error level should be actionable in that the tracebacks or logged errors should allow resolution of the error with a code or configuration update. These errors are generally considered failures of the program to anticipate corner conditions or failure modes.

warning

An expected error condition or fault in inputs to which the application responds correctly, but the end-user action may not be what they intended. Incorrect passwords, or actions that are not allowed because they conflict with existing configurations are appropriate for this level.

Errors logged at an warning level may not be actionable, but should be informative in the logs to indicate what the failure was. Errors where secure information are part of the response may include more information in logs than in a response ot the end user for security considerations.

info

Informational data about current execution that would be relevant to regular use of the application. Not generally considered “errors” at the log level of info, this level should be used judiciously with the idea that regular operation of the application is likely to run with log filtering set to allow info logging.

Information logged at the info is not expected to be actionable, but may be expected to be used in external systems collecting the log information for regular operational metrics.

debug

Informational data about current execution that would be relevant to debugging or detailed analysis of the application, typically for a programmer, or to generate logs for post-analysis by a someone familiar with the code in the project. Information is not considered “errors” at the log level of debug.

Information logged at the debug is not expected to be actionable, but may be expected to be used in external systems collecting the log information for debugging or post-analysis metrics.

Setting up and using Logging¶

Using our dependency injection libraries, it’s typical to inject Logger and then use it within appropriate methods. Within factory methods for services or modules, Logger is initialized with the module name, which annotates the logs with information about where the logs were coming from.

An example of this:

di.annotate(someFactory, new di.Inject('Logger'))

function someFactory (Logger) {
    var logger = Logger.initialize(someFactory);
}

with logger being used later within the relevant scope for logging. For example:

function foo(bar, baz) {
    logger.debug("Another request was made with ", {id: baz});
}

The definitions for the methods and what the code does can be found in the logger module.

Deprecation¶

There is a special function in our logging common library for including in methods you’re attempting to deprecate:

logger.deprecate("This shouldn't be used any longer", 2)

Which will generate log output at the error for assistance in identifying methods, APIs, or subsystems that are still in use but in the process of being depracted for replacement.

Discovery with a Default Workflow¶

Sequence Diagram for the Discovery Workflow

The diagram is made with WebSequenceDiagrams.

To see if the DHCP request was received by ISC DHCP, look in /var/log/syslog of the RackHD host. grep DHCP /var/log/syslog works reasonably well - you’re looking for a sequence like this:

Jan  8 15:43:43 rackhd-demo dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 3 (xid=0x5b3b9260)
Jan  8 15:43:43 rackhd-demo dhclient: DHCPREQUEST of 10.0.2.15 on eth0 to 255.255.255.255 port 67 (xid=0x60923b5b)
Jan  8 15:43:43 rackhd-demo dhclient: DHCPOFFER of 10.0.2.15 from 10.0.2.2
Jan  8 15:43:43 rackhd-demo dhclient: DHCPACK of 10.0.2.15 from 10.0.2.2

You should also see the DHCP proxy return the bootfile. In the DHCP-proxy logs, look for lines with DHCP.messageHandler:

S 2016-01-08T19:31:43.268Z [on-dhcp-proxy] [DHCP.messageHandler] [Server] Unknown node 08:00:27:f3:9f:2e. Sending down default bootfile.

And immediately thereafter, you should see the server request the file from TFTP:

S 2016-01-08T19:31:43.352Z [on-tftp] [Tftp.Server] [Server] tftp: 67.300 monorail.ipxe

Default discovery workflow¶

title Default Discovery Workflow
Server->RackHD: DHCP from PXE(nic or BIOS)
RackHD->Server: ISC DHCP response with IP
note over RackHD:
    If the node is already "known",
    it will only respond if there's an active workflow
    that's been invoked related to the node
end note
RackHD->Server: DHCP-proxy response with bootfile
Server->RackHD: Request to download bootfile via TFTP
RackHD->Server: TFTP sends requested file (monorail.ipxe)
note over Server:
    Server loads monorail.ipxe
    and initiates on bootloader
end note
Server->RackHD: IPXE script requests what to do from RackHD (http)
note over RackHD:
    RackHD looks up IP address of HTTP request from iPXE script to find the node via its mac-address.
    1) If the node is already "known", it will only respond if there's an active workflow
    that's been invoked related to the node.
    2) If the node isn't known, it will create a workflow (default is the workflow 'Graph.Sku.Discovery')
    and respond with an iPXE script to initiate that.
end note
RackHD->Server: iPXE script (what RackHD calls a Profile) (via http)
note over Server:
    iPXE script with RancherOS vmlinuz,
    initrd and cloud-config (http)
end note
Server->RackHD: iPXE requests static file - the RancherOS vmlinuz kernel
RackHD->Server: RancherOS vmlinuz (http)
Server->RackHD: iPXE requests static file - RacherOS initrd
RackHD->Server: RancherOS initrd (http)
note over Server:
    Server loads the vmlinuz and initrd,
    and transfers control (boots RancherOS)
end note
Server->RackHD: RancherOS requests cloud-config - RacherOS cloud-config
RackHD->Server: RancherOS cloud-config(http)
Server->RackHD: RancherOS loads discovery docker image from Server
note over Server:
    the discovery container is set to request
    and launch a NodeJS task runnner
end note
Server->RackHD: requests the bootstrap.js template
RackHD->Server: bootstrap.js filled out with values specific to the node based on a lookup
note over Server:
    runs node bootstrap.js
end note
Server->RackHD: bootstrap asks for tasks (what should I do?)
RackHD->Server: data packet of tasks (via http)
note over Server:
    Discovery Workflow
    passes down tasks to
    interrogate hardware
end note
loop for each Task from RackHD
    Server->RackHD: output of task
end
note over RackHD
    Task output stored as catalogs in RackHD related to the node.
    If RackHD is configured with SKU definitions,
    it processes these catalogs to determine the SKU.
    If there's a SKU specific workflow defined, control is continued to that.
    The discovery workflow will create an enclosure node based on the catalog data.
    The discovery workflow will also create IPMI pollers for the node,
    if relevent information can be found in the catalog.
    The discovery workflow will also generate tag for the node,
    based on user-defined tagging rules.
end note
Server->RackHD: bootstrap asks for tasks (what should I do?)
RackHD->Server: Nothing more, thanks - please reboot (via http)

Footprint Benchmark Test¶

Footprint benchmark test collects system data when running poller (15min), node discovery and CentOS bootstrap test cases. It can also run independently from any test cases, allowing users to measure footprint about any operations they carry out. The data includes CPU, memory, disk and network consumption of every process in RackHD, as well as RabbitMQ and MongoDB processes. The result is presented as HTML files. For more details, please check the wiki page proposal-footprint-benchmarks.

How It Works¶

Footprint benchmark test is integrated into RackHD test framework. It can be executed as long as the machine running the test can access the RackHD API and manipulate the RackHD machine via SSH.

sudo socat -d -d TCP4-LISTEN:55672,reuseaddr,fork TCP4:localhost:5672

git clone https://github.com/RackHD/RackHD.git

cd RackHD/test
virtualenv .venv
source .venv/bin/activate
pip install -r requirements.txt

vim config/config.ini

python benchmark.py

python benchmark.py --group=poller|discovery|bootstrap

python benchmark.py --start|stop

python benchmark.py --getdir

chrome.exe --user-data-dir="C:/Chrome dev session" --allow-file-access-from-files

Logged warnings FAQ¶

Question:

I’m seeing this warning appear in the logs but it all seems to be working. What’s happening?

W 2016-01-29T21:06:22.756Z [on-tftp] [Tftp.Server] [Server] Tftp error
 -> /lib/server.js:57
file:          monorail.ipxe
remoteAddress: 172.31.128.5
remotePort:    2070
W 2016-01-29T21:12:43.783Z [on-tftp] [Tftp.Server] [Server] Tftp error
 -> /lib/server.js:57
file:          monorail.ipxe
remoteAddress: 172.31.128.5
remotePort:    2070

Answer:

What I learned (so I may be wrong here, but think it’s accurate) is that during the boot loading/PXE process the NICs will attempt to interact with TFTP in such a way that the first request almost always fails - it’s how the C code in those nics is negotiating for talking with TFTP. So you’ll frequently see those errors in the logs, and then immediately also see the same file downloading on the second request from the nic (or host) doing the bootloading.

Question:

When we’re boostraping a node (or running a workflow against a node in general) with a NUC, we sometimes see these extended messages on the server’s console reading Link...... down, and depending on the network configuration can see failures for the node to bootstrap and respond to PXE.

Answer:

The link down is a pernicious problem for PXE booting in general, and a part of the game that’s buried into how switches react and bring up and down ports. We’ve generally encouraged settings like “portfast” which more agressively bring up links that are going down and coming back up with a power cycle. In the NUCs you’re using, you’ll see that extensively, but it happens on all networks. If you have spanning-tree enabled, some things like that - it’ll expand the time. There’s only so much we can do to work around it, but fundamentally it means that while the relevant computer things things are “UP and OK” and has started a TFTP/PXE boot process, the switch hasn’t brought the NIC link up. So we added an explicit sleep in there in the monorail.ipxe to extend ‘the time to let networks converge so that the process has a better chance of succeeding.

Security Constraints¶

RackHD as a technology is configured to control and automate hardware, which implies a number of natural security concerns. As a service, it provides an API control endpoint, which in turn uses protocols on networks relevant to the hardware it’s managing. One of the most common of those protocols is IPMI, which has known security flaws, but is used because it’s one of the most common mechanisms to control datacenter servers.

A relatively common requirement in datacenters is that networks used for IPMI traffic are isolated from other networks, to limit the vectors by which IPMI endpoints could be attacked. When RackHD is using IPMI, it simply needs to have L3 (routed IP) network traffic to the relevant endpoints in order for the workflow engine and various controls to operate.

Access to IPMI endpoints on hardware can be separated off onto it’s own network, or combined with other networks. It is generally considered best practice to separate this network entirely, or constrain it to highly controlled networks where access is strictly limited.

Hardware Controls¶

insmod: ERROR: could not load module /opt/drivers/ipmi_msghandler.ks: No such file or directory

RackHD manages hardware generally using at least one network interface. Network switches typically have an administrator network interface, and Smart PDUs that can be managed by RackHD have a administrative gateway.

Compute servers have the most varied and complex setup, with data center servers often leveraging a BMC (Baseboard Management Controller). A BMC is a separate embedded computer monitoring and controlling a larger computer. The protocol used most commonly to communicate to a BMC is IPMI, the details of which can matter significantly.

Desktop class machines (and many laptops) often do not have BMCs, although some Intel desktops may have an alternative technology: AMT which provides some similiar mechanisms.

You can view a detailed diagram of the components inside a BMC at IPMI Basics, although every hardware vendor is slighty different in how they configure their servers. The primary difference for most Intel-based server vendors is how the BMC network interface is exposed. There are two options that you will commonly see:

• LOM : Lights out Management

  The BMC has has a dedicated network interface to the BMC

• SOM : “Shared on motherboard”

  The network interface to the BMC shares a network interface with the motherboard. In these cases, the same physical plug is backed by two internal network interfaces (each with its own hardware address).

If you’re working with a server with a network interface shared by the motherboard and BMC, then separating the networks that provide IPMI access and the networks that the server will use during operation may be significantly challenging.

The BMC provides a lot of information about the computer, but not everything. Frequently devices such as additional NIC cards, RAID array controllers, or other devices attached to internal PCI busses aren’t accessible or known about from the BMC. This is why RackHD’s default discovery mechanism operates by Discovery and Geneaology, which loads an OS into RAM on the server and uses that OS to interrogate the hardware.

IP Address Management¶

With multiple networks in use with RackHD, how machines are getting IP addresses and what systems are repsonsible for providing those IP addresses another critical concern. Running DHCP, which RackHD integrates with tightly to enable PXE booting of hosts, much be done carefully and there should only ever be a single DHCP server running on a given layer-2 network. Many existing systems will often already have DHCP servers operational or a part of their environment, or may mandate that IP addresses are set statically or provided via a static configuration.

RackHD can be configured without a local DHCP instance, although DHCP is a required component for PXE booting a host. If DHCP is provided externally, then RackHD only needs to provide the on-dhcp-proxy process, which will need to be on the same network as the DHCP server, and leverages the DHCP protocols capability to separate out the service providing the TFTP boot information from the service providing IP address (and other) configuration details for hosts.

RackHD Network Access Requirements¶

• DHCP-proxy

  The DHCP proxy service for RackHD needs to be on the same Layer 2 (broadcast) network as DHCP to provide PXE capabilities to machines PXE booting on that network.

• TFTP, HTTP

  The PXE network also needs to be configured to expose the south-bound HTTP API interfaces from on-http and the on-tftp service to support RackHD PXE booting hosts by providing the bootloaders, and responding to requests for files and custom templates or scripts that coordinate with RackHD’s workflow engine.

• IPMI, HTTP/Redfish, SNMP

  Layer 3 (routed IP) access to the out of band network - the network used to communicate with server BMCs, SmartPDU management gateways, or Network switch administrative network interfaces.

Possible Configurations¶

In an environment where the hardware you’re managing doesn’t have additional network interfaces, and the BMC shares the motherboard physical network interface, the configuration will be fairly limited.

In this example, RackHD is providing DHCP to a network which is connected through a layer3 switch or router to the rest of the network. RackHD’s DHCP server can provide IP addresses to the motherboard NICs as the PXE boot, and may also provide IP addresses to the BMCs if they are configured to use DHCP.

If the compute servers are not configured to use DHCP in this setup, then the BMC IP addresses must be statically set/assigned and carefully managed so as to not overlap with the DHCP range that RackHD’s DHCP services are providing.

In this example, the servers have a dedicated “lights out” network interface, which is on a separate network and RackHD can access it via one of its interfaces. RackHD is still providing DHCP to the servers for PXE booting on the motherboard, but the IP addresses of the BMCs can be completely indepdent in how they are provided.

This example, or a variation on it, is how you might configure a RackHD deployment in a dedicated data center where the same people responsible for running RackHD are responsible for the IP addresses and general datacenter infrastructure. In general, this kind of configuration is what you might do with shared responsibilities and close coordination between network configurations within and external to RackHD

In this example, all the networks are isolated and separate, and in this case isolated to the instance of RackHD as well. RackHD may be multiple network interfaces assigned to it with various network configurations. The BMC network can be set to use a DHCP or statically assigned IP addresses - as long as the network routing is clear and consistent to RackHD. The servers also have multiple network interface cards attached to the motherboard, each of which can be on separate networks, or they can be used in combined configurations.

This example highlights how RackHD might be configured if it was being used to independently manage a rack of gear, as in an “rack of machines as an appliance” use case, or in a very large scale environment, where every rack has it’s own dedicated management network that are functionally identical.

Configuration Parameters¶

The following table describes the configuration parameters in config.json:

"amqp": "amqp[s]://localhost",
"amqp": "amqp[s]://<host>:<port>",

"amqp": ["amqp[s]://<host_1>:<port_1>","amqp[s]://<host_2>:<port_2>",..., "amqp[s]://<host_n:<port_n>"],

{
    "enabled": true,
    "keyFile": "/path/to/key/file",
    "certFile": "/path/to/cert/file",
    "caFile": "/path/to/cacert/file"
}

openssl enc -aes-256-cbc -k secret -P -md sha1

These configurations can also be overridden by setting environment variables in the process that’s running each application, or on the command line when running node directly. For example, to override the value of amqp for the configuration, you could use:

export amqp=amqp://another_host:5763

prior to running the relevant application.

HTTPS/TLS Configuration¶

To use TLS, a private RSA key and X.509 certificate must be provided. On Ubuntu and Mac OS X, the openssl command line tool can be used to generate keys and certificates.

For internal development purposes, a self-signed certificate can be used. When using a self-signed certificate, clients must manually include a rule to trust the certificate’s authenticity.

By default, the application uses a self-signed certificate issued by Monorail which requires no configuration. Custom certificates can also be used with some configuration.

Parameters

See the table in Configuration Parameters for information about HTTP/HTTPS configuration parameters. These parameters beging with HTTP and HTTPS.

BMC Username and Password Configuration¶

A node gets discovered and the BMC IPMI comes up with a default username/password. User can automatically set IPMI OBM settings using a default user name(‘__rackhd__’) and an auto generated password in rackHD by adding the following to RackHD config.json:

"autoCreateObm": "true"

If a user wants to change the BMC credentials later in time, when the node has been already discovered and database updated, a separate workflow located at on-taskgraph/lib/graphs/bootstrap-bmc-credentials-setup-graph.js can be posted using Postman or Curl command.

POST: http://server-ip:8080/api/current/workflows/

add the below content in the json body for payload (example node identifier and username, password shown below)

{
    "name": "Graph.Bootstrap.With.BMC.Credentials.Setup",
    "options": {
         "defaults": {
             "graphOptions": {
                 "target": "56e967f5b7a4085407da7898",
                 "generate-pass": {
                     "user": "7",
                     "password": "7"
                 }
             },
             "nodeId": "56e967f5b7a4085407da7898"
         }
     }
}

By running this workflow, a boot-graph runs to bootstrap an ubuntu image on the node again and set-bmc-credentials-graph runs the required tasks to update the BMC credentials. Below is a snippet of the ‘Bootstrap-And-Set-Credentials graph’, when the graph is posted the node reboots and starts the discovery process

 module.exports = {
   friendlyName: 'Bootstrap And Set Credentials',
   injectableName: 'Graph.Bootstrap.With.BMC.Credentials.Setup',
   options: {
       defaults: {
           graphOptions: {
               target: null
           },
           nodeId: null
       }
   },
   tasks: [
       {
           label: 'boot-graph',
           taskDefinition: {
               friendlyName: 'Boot Graph',
               injectableName: 'Task.Graph.Run.Boot',
               implementsTask: 'Task.Base.Graph.Run',
               options: {
                   graphName: 'Graph.BootstrapUbuntu',
                   defaults : {
                       graphOptions: {   }
                   }
               },
               properties: {}
           }
       },
       {
           label: 'set-bmc-credentials-graph',
           taskDefinition: {
               friendlyName: 'Run BMC Credential Graph',
               injectableName: 'Task.Graph.Run.Bmc',
               implementsTask: 'Task.Base.Graph.Run',
               options: {
                   graphName: 'Graph.Set.Bmc.Credentials',
                   defaults : {
                       graphOptions: {   }
                   }
               },
               properties: {}
           },
           waitOn: {
               'boot-graph': 'finished'
           }
       },
       {
           label: 'finish-bootstrap-trigger',
           taskName: 'Task.Trigger.Send.Finish',
           waitOn: {
               'set-bmc-credentials-graph': 'finished'
           }
       }
   ]
};

To remove the BMC credentials, User can run the following workflow located at on-taskgraph/lib/graphs/bootstrap-bmc-credentials-remove-graph.js and can be posted using Postman or Curl command.

POST: http://server-ip:8080/api/current/workflows/

add the below content in the json body for payload (example node identifier and username, password shown below)

{
    "name": "Graph.Bootstrap.With.BMC.Credentials.Remove",
    "options": {
         "defaults": {
             "graphOptions": {
                 "target": "56e967f5b7a4085407da7898",
                 "remove-bmc-credentials": {
                     "users": ["7","8"]
                 }
             },
             "nodeId": "56e967f5b7a4085407da7898"
         }
     }
}

Certificates¶

This section describes how to generate and install a self-signed certificate to use for testing.

openssl genrsa -out privkey.pem 2048

openssl req -new -x509 -key privkey.pem -out cacert.pem -days 9999

mv privkey.pem /opt/monorail/data/mykey.pem
mv cacert.pem /opt/monorail/data/mycert.pem

Setup HTTP/HTTPS endpoint¶

This section describes how to setup HTTP/HTTPS endpoints in RackHD. An endpoint is an instance of HTTP or HTTPS server that serves a group of APIs. Users can choose to enable authentication or enable HTTPS for each endpoint.

There is currently one API group defined in RackHD:

• the northbound-api-router API group. This is the API group that is used by users

[
    {
        "address": "0.0.0.0",
        "port": 8443,
        "httpsEnabled": true,
        "httpsCert": "data/dev-cert.pem",
        "httpsKey": "data/dev-key.pem",
        "httpsPfx": null,
        "proxiesEnabled": false,
        "authEnabled": false,
        "yamlName": ["monorail-2.0.yaml", "redfish.yaml"]
    }
]

Setup Taskgraph Endpoint¶

This section describes how to setup the taskgraph endpoint in RackHD. The taskgraph endpoint is the interface that is used by nodes to interacting with the system

"taskGraphEndpoint": {
    "address": "172.31.128.1",
    "port": 9030
}

Raid Configuration¶

sudo apt-get install alien
sudo alien -k perccli-1.11.03-1.noarch.rpm

#This creates the dell.raid.docker.tar.xz image
cd on-imagebuilder/oem/dell-raid
sudo docker build -t rackhd/micro .
sudo docker save rackhd/micro | xz -z > dell.raid.docker.tar.xz

#This creates the raid.docker.tar.xz image
cd on-imagebuilder/oem/raid
sudo docker build -t rackhd/micro .
sudo docker save rackhd/micro | xz -z > raid.docker.tar.xz

{
   "options": {
       "config-raid":{
                  "ssdStoragePoolArr":[],
                  "ssdCacheCadeArr":[{
                         "enclosure": 252,
                         "type": "raid0",
                         "drives":"[0]"
                  }],
                  "controller": 0,
                  "path":"/opt/MegaRAID/storcli/storcli64",
                  "hddArr":[{
                          "enclosure": 252,
                          "type": "raid0",
                          "drives":"[1]"
                   },
                   {
                         "enclosure": 252,
                         "type": "raid1",
                         "drives":"[4,5]"
                   }]
      }
   }
}

{
    label: 'refresh-catalog-megaraid',
    taskName: 'Task.Catalog.megaraid',
    waitOn: {
        'config-raid': 'succeeded'
    }
 },
 {
    label: 'final-reboot',
    taskName: 'Task.Obm.Node.Reboot',
    waitOn: {
        'refresh-catalog-megaraid': 'finished'
    }
 }

monorail@monorail-micro:~$ sudo /opt/MegaRAID/storcli/storcli64 /c0/vall show Virtual Drives : ==============-------------------------------------------------------------- DG/VD TYPE State Access Consist Cache Cac sCC Size Name ---------------------------------------------------------------
0/0 Cac0 Optl RW Yes NRWBD - ON 372.093 GB
1/1 RAID0 Optl RW Yes RWTD - ON 1.090 TB
2/2 RAID0 Optl RW Yes RWTD - ON 1.090 TB
3/3 RAID0 Optl RW Yes RWTD - ON 1.090 TB
4/4 RAID0 Optl RW Yes RWTD - ON 1.090 TB
5/5 RAID0 Optl RW Yes RWTD - ON 1.090 TB

Starting and Stopping the API Server¶

The API server runs by default. Use the following commands to stop or start the API server.

Generating API Documentation¶

You can generate an HTML version of the API documentation by cloning the on-http repository and running the following command.

$ git clone https://github.com/RackHD/on-http
$ cd on-http
$ npm install
$ npm run apidoc
$ npm run taskdoc

The default and example quick start build that we describe in ../tutorials/vagrant has the API docs rendered and embedded within that instance for easy use, available at http://[IP ADDRESS OF VM]:8080/docs/ for the 1.1 API documentation, and http://[IP ADDRESS OF VM]:8080/swagger-ui/ for the current (2.0) and Redfish API documentation.

RackHD Client Libraries¶

The 2.0 API generates a swagger API definition file that can be used to create client libraries with swagger. To create this file locally, you can check out the on-http library and run the commands:

npm install
npm run apidoc

The resulting files will be in build/swagger-doc and will be pdf files that are documentation for the 2.0 API (rackhd-api-2.1.0.pdf) and the Redfish API (rackhd-redfish-v1-1.1.1.pdf).

To create a client library you can run the command:

npm run client -- -l <language>

Where the language you input can currently be python, go, or java. Go is generated using go-swagger and python and java are generated using swagger-codegen. This command will generate client libraries for the 2.0 API and Redfish API and will be in the saved in the directories on-http/on-http-api2.0` and ``on-http/on-http-redfish-1.0 , respectively.

You can also use the swagger generator online tool to generate a client zip bundle for a variety of languages, including python, Java, javascript, ruby, scala, php, and more.

Examples using the python client library¶

Getting a list of nodes

from on_http import NodesApi, ApiClient, Configuration

    config = Configuration()
    config.debug = True
    config.verify_ssl = False

    client = ApiClient(host='http://localhost:9090',header_name='Content-Type',header_value='application/json')
    nodes = NodesApi(api_client=client)
    nodes.api2_0_nodes_get()
    print client.last_response.data

Deprecated 1.1 API - Getting a list of nodes:

from on_http import NodesApi, ApiClient, Configuration

config = Configuration()
config.debug = True
config.verify_ssl = False

client = ApiClient(host='http://localhost:9090',header_name='Content-Type',header_value='application/json')
nodes = NodesApi(api_client=client)
nodes.api1_1_nodes_get()
print client.last_response.data

Or the same asynchronously (with a callback):

def cb_func(resp):
print 'GET /nodes callback!', resp

thread = nodes.api2_0_nodes_get(callback=cb_func)

Deprecated 1.1 API - Or the same asynchronously (with a callback):

def cb_func(resp):
print 'GET /nodes callback!', resp

thread = nodes.api1_1_nodes_get(callback=cb_func)

Using Pagination¶

The RackHD 2.0 /nodes, /pollers, and /workflows APIs support pagination using $skip and $top query parameters.

These parameters can be used individually or combined to display any subset of consecutive resources in the collection.

Here is an example request using $skip and $top to get get the second page of nodes with four items per page.

curl http://localhost:8080/api/current/nodes?$skip=4&$top=4

RackHD will add a link header to assist in traversing a large collection. Links will be added if either $skip or $top is used and the size of the collection is greater than the number of resources displayed (i.e. the collection cannot fit on one page). If applicable, links to first, last, next, and previous pages will be included in the header. The next and previous links will be omitted for the last and first pages respectively.

Here is an example link header from a collection containing 1000 nodes.

</api/current/nodes?$skip=0&$top=4>; rel="first",
</api/current/nodes?$skip=1004&$top=4>; rel="last",
</api/current/nodes?$skip=0&$top=4>; rel="prev",
</api/current/nodes?$skip=8&$top=4>; rel="next"

Events Payloads¶

All published external events’ payload formats are common, the event attributes are as below:

The table of type, typeId, action and severity for all external events

Example of heartbeat event payload:

{
    "version": "1.0",
    "type": "heartbeat",
    "action": "updated",
    "typeId": "kickseed.example.com.on-taskgraph",
    "severity": "information",
    "createdAt": "2016-07-13T14:23:45.627Z",
    "nodeId": "null",
    "data": {
        "name": "on-taskgraph",
        "title": "node",
        "pid": 6086,
        "uid": 0,
        "platform": "linux",
        "release": {
            "name": "node",
            "lts": "Argon",
            "sourceUrl": "https://nodejs.org/download/release/v4.7.2/node-v4.7.2.tar.gz",
            "headersUrl": "https://nodejs.org/download/release/v4.7.2/node-v4.7.2-headers.tar.gz"
        },
        "versions": {
            "http_parser": "2.7.0",
            "node": "4.7.2",
            "v8": "4.5.103.43",
            "uv": "1.9.1",
            "zlib": "1.2.8",
            "ares": "1.10.1-DEV",
            "icu": "56.1",
            "modules": "46",
            "openssl": "1.0.2j"
        },
        "memoryUsage": {
            "rss": 116531200,
            "heapTotal": 84715104,
            "heapUsed": 81638904
        },
        "currentTime": "2017-01-24T07:18:49.236Z",
        "nextUpdate": "2017-01-24T07:18:59.236Z",
        "lastUpdate": "2017-01-24T07:18:39.236Z",
        "cpuUsage": "NA"
    }
}

Example of node discovered event payload:

{
    "type": "node",
    "action": "discovered",
    "typeId": "58aa8e54ef2b49ed6a6cdd4c",
    "nodeId": "58aa8e54ef2b49ed6a6cdd4c",
    "severity": "information",
    "data": {
        "ipMacAddresses": [
            {
                "ipAddress": "172.31.128.2",
                "macAddress": "2c:60:0c:ad:d5:ba"
            },
            {
                "macAddress": "90:e2:ba:91:1b:e4"
            },
            {
                "macAddress": "90:e2:ba:91:1b:e5"
            },
            {
                "macAddress": "2c:60:0c:c0:a8:ce"
            }
        ],
        "nodeId": "58aa8e54ef2b49ed6a6cdd4c",
        "nodeType": "compute"
    },
    "version": "1.0",
    "createdAt": "2017-02-20T06:37:23.775Z"
}

Events via AMQP¶

heartbeat.updated.information.kickseed.example.com.on-tftp

polleralert.sel.updated.critical.44b15c51450be454180fabc.57b15c51450be454180fa460

node.discovered.information.57b15c51450be454180fa460.57b15c51450be454180fa460

graph.started.information.35b15c51450be454180fabd.57b15c51450be454180fa460

$ sudo node sniff.js "on.events" "heartbeat.#"

$ sudo node sniff.js "on.events" "#.discovered.#"

Events via Hook¶

curl -H "Content-Type: application/json" -X POST -d @payload.json <server>api/current/hooks

{
    "url": "http://www.abc.com/def"
}

{
    "name": "event sets",
    "url": "http://www.abc.com/def",
    "filters": [
        {
            "type": "node",
            "nodeId": "57b15c51450be454180fa460"
        },
        {
            "type": "node",
            "action": "discovered|updated",
        }
    ]
}

POST /api/2.0/hooks
{
    "url": "http://www.abc.com/def"
}

DELETE /api/2.0/hooks/:id

GET /api/2.0/hooks

GET /api/2.0/hooks/:id

PATCH /api/2.0/hooks/:id
{
    "name": "New Hook"
}

Redfish Alert Notification¶

POST /redfish/v1/EventService/Subscriptions
    {
            "Context": "context string",
            "Description": "Event Subscription Details",
            "Destination": "https://10.240.19.226:8443/api/2.0/notification/alerts",
            "EventTypes": [
            "ResourceAdded",
            "StatusChange",
                "Alert"
            ],
            "Id": "id",
            "Name": "name",
            "Protocol": "Redfish"
    }

{
  "@odata.context": "/redfish/v1/$metadata#EventDestination.EventDestination",
  "@odata.id": "/redfish/v1/EventService/Subscriptions/b50106d4-32c6-11e7-8b05-64006ac35232",
  "@odata.type": "#EventDestination.v1_0_2.EventDestination",
  "Context": "RackhHD Subscription",
  "Description": "Event Subscription Details",
  "Destination": "https://10.1.1.1:8443/api/2.0/notification/alerts",
  "EventTypes": [
    "ResourceAdded",
    "StatusChange",
    "Alert"
  ],
  "EventTypes@odata.count": 3,
  "Id": "b50106d4-32c6-11e7-8b05-64006ac35232",
  "Name": "EventSubscription b50106d4-32c6-11e7-8b05-64006ac35232",
  "Protocol": "Redfish"
}

{
    "options": {
            "redfish-subscribtion": {
                    "url": "https://10.240.19.130/redfish/v1/EventService/Subscriptions",
                    "credential": {
                            "username": "root",
                            "password": "1234567"
                    },
                    "data": {
                            "Context": "context string",
                            "Description": "Event Subscription Details",
                            "Destination": "https://1.1.1.1:8443/api/2.0/notification/alerts",
                            "EventTypes": [
                                    "StatusChange",
                                    "Alert"
                            ],
                            "Id": "id",
                            "Name": "name",
                            "Protocol": "Redfish"
                    }

            }
    }
}

{
        "type": "node",
        "action": "alerts",
        "data": {
                "Context": "context string",
                "EventId": "8689",
                "EventTimestamp": "2017-04-03T10:07:32-0500",
                "EventType": "Alert",
                "MemberId": "7e675c8e-127a-11e7-9fc8-64006ac35232",
                "Message": "The coin cell battery in CMC 1 is not working.",
                "MessageArgs": ["1"],
                "MessageArgs@odata.count": 1,
                "MessageId": "CMC8572",
                "Severity": "Critical",
                "sourceIpAddress": "10.240.19.130",
                "nodeId": "58d94cec316779d4126be134",
                "sourceMacAddress   ": "64:00:6a:c3:52:32",
                "ChassisName": "PowerEdge R630",
                "ServiceTag": "4666482",
                "SN": "CN747515A80855"
        },
        "severity": "critical",
        "typeId": "58d94cec316779d4126be134",
        "version": "1.0",
        "createdAt": "2017-04-03T14:11:46.245Z"
}

Defining Nodes¶

Nodes are defined via a JSON definition that conform to this schema:

• id (string): unique identifier for the node
• type (string): a human readable name for the graph
• name (string): a unique name used by the system and the API to refer to the graph
• autodiscover (boolean):
• sku (string): the SKU ‘id’ that has been matched from the SKU workflow task
• createdAt (string): ISO8601 date string of time resource was created
• updatedAt (string): ISO8601 date string of time resource was last updated
• identifiers (array of strings): a list of strings that make up alternative identifiers for the node
• obms (array of objects): a list of objects that define out-of-band management access mechanisms
• relations (array of objects): a list of relationship objects

Defining Catalogs¶

• id (string): unique identifier for the node
• createdAt (string): ISO8601 date string of time resource was created
• updatedAt (string): ISO8601 date string of time resource was last updated
• data (json): A JSON data structure specific to the catalog tool
• node (string): the node to which this catalog is associated
• source (string): type of the data

API Commands for Nodes¶

The following are common API commands that can be used when running the on-http process.

Get Nodes

GET /api/current/nodes

curl <server>/api/current/nodes

Get Specific Node

GET /api/current/nodes/<id>

curl <server>/api/current/nodes/<id>

Sample switch node after Discovery

{
    "type":"switch",
    "name":"nodeName",
    "autoDiscover":true,
    "service": "snmp-ibm-service",
    "config": {
        "host": "10.1.1.3"
    },
    "createdAt":"2015-07-27T22:03:45.353Z",
    "updatedAt":"2015-07-27T22:03:45.353Z",
    "id":"55b6aac1024fd1b349afc145"
}

Sample compute node after Discovery

{
    "autoDiscover": false,
    "catalogs": [],
    "createdAt": "2015-11-30T21:37:18.441Z",
    "id": "565cc18ec3f522fe51620fa2",
    "identifiers": [
        "08:00:27:27:eb:12"
    ],
    "name": "08:00:27:27:eb:12",
    "obms": [
        {
            "ref": "/api/2.0/obms/58806bb776fab9d82b831e52",
            "service": "noop-obm-service"
        }
    ],
    "relations": [
        {
            "relationType": "enclosedBy",
            "targets": [
                "565cc1d2807f92fc51a7c9c5"
            ]
        }
    ],
    "sku": "565cb91669aa70ab450da9dd",
    "type": "compute",
    "updatedAt": "2015-11-30T21:38:26.755Z",
    "workflows": []
}

List all the (latest) catalog data associated with a node

GET /api/current/nodes/<id>/catalogs

curl <server>/api/current/nodes<id>/catalogs

To retrieve a specific catalog source for a node

GET /api/current/nodes/<id>/catalogs/<source>

curl <server>/api/current/nodes<id>/catalogs/<source>

Sample Output:

{
    "createdAt": "2015-11-30T21:37:49.696Z",
    "data": {
        "BIOS Information": {
            "Address": "0xE0000",
            "Characteristics": [
                "ISA is supported",
                "PCI is supported",
                "Boot from CD is supported",
                "Selectable boot is supported",
                "8042 keyboard services are supported (int 9h)",
                "CGA/mono video services are supported (int 10h)",
                "ACPI is supported"
            ],
            "ROM Size": "128 kB",
            "Release Date": "12/01/2006",
            "Runtime Size": "128 kB",
            "Vendor": "innotek GmbH",
            "Version": "VirtualBox"
        },
        "Base Board Information": {
            "Asset Tag": "Not Specified",
            "Chassis Handle": "0x0003",
            "Contained Object Handles": "0",
            "Features": [
                "Board is a hosting board"
            ],
            "Location In Chassis": "Not Specified",
            "Manufacturer": "Oracle Corporation",
            "Product Name": "VirtualBox",
            "Serial Number": "0",
            "Type": "Motherboard",
            "Version": "1.2"
        },
        "Chassis Information": {
            "Asset Tag": "Not Specified",
            "Boot-up State": "Safe",
            "Lock": "Not Present",
            "Manufacturer": "Oracle Corporation",
            "Power Supply State": "Safe",
            "Security Status": "None",
            "Serial Number": "Not Specified",
            "Thermal State": "Safe",
            "Type": "Other",
            "Version": "Not Specified"
        },
        "Inactive": [
            {},
            {},
            {}
        ],
        "OEM Strings": {
            "String 1": "vboxVer_5.0.10",
            "String 2": "vboxRev_104061"
        },
        "OEM-specific Type": {
            "Header and Data": [
                "80 08 08 00 E7 7D 21 00"
            ]
        },
        "System Information": {
            "Family": "Virtual Machine",
            "Manufacturer": "innotek GmbH",
            "Product Name": "VirtualBox",
            "SKU Number": "Not Specified",
            "Serial Number": "0",
            "UUID": "992DA874-C028-4CDD-BB06-C86D525A7056",
            "Version": "1.2",
            "Wake-up Type": "Power Switch"
        }
    },
    "id": "565cc1ad807f92fc51a7c9bf",
    "node": "565cc18ec3f522fe51620fa2",
    "source": "dmi",
    "updatedAt": "2015-11-30T21:37:49.696Z"
}

Out of Band Management Settings¶

Get list of Out of Band Management settings that have been associated with nodes.

Get list of OBMs settings

GET /api/current/obms

curl <server>/api/current/obms

Get list of OBMs schemas showing required properties to create an OBM

GET /api/current/obms/definitions

curl <server>/api/current/obms/definitions

Create or update a single OBM service and associate it with a node

PUT /api/current/obms

curl -X PUT -H "Content-Type: application/json" -d '{ "nodeId": <node id>, "service": "ipmi-obm-service", "config": { "user": "admin", "password": "admin", "host": "<host ip>" } }' /api/current/obms

Example output of PUT

{
  "id": "5911fa6447f8b7b207f9a485",
  "node": "/api/2.0/nodes/590cbcbf29ba9e40471c9f3c",
  "service": "ipmi-obm-service",
  "config": {
    "user": "admin",
    "host": "172.31.128.2"
  }
}

Get a specific OBM setting

GET /api/current/obms/<id>

curl <server>/api/current/obms/<id>

PATCH an OBM setting

PATCH /api/current/obms/<id>

curl -X PUT -H "Content-Type: application/json" -d '{ "nodeId": <node id>, "service": "ipmi-obm-service", "config": { "user": "admin", "password": "admin", "host": "<host ip>" } }' /api/current/obms/<id>

Delete an OBM setting

DELETE /api/current/obms/<id>

curl -X DELETE <server>/api/current/obms/<id>

To set a no-op OBM setting on a node

curl -X PUT -H "Content-Type:application/json" localhost/api/current/nodes/5542b78c130198aa216da3ac -d '{  { "service": "noop-obm-service", "config": { } } }'

To set a IPMI OBM setting on a node

curl -X PUT -H 'Content-Type: application/json' -d ' { "service": "ipmi-obm-service", "config": { "host": "<host ip>", "user": "admin", "password": "admin" } }' <server>/api/current/nodes/<nodeID>/obm

How to use obms when more than one obm are present on a node

Example: when update firmware workflow is called on a node that has multiple obms (ipmi-obm-service, redfish-obm-service), the payload needs to call out what obm service to use for certain tasks within the workflow that use the obm service..

POST /api/current/nodes/<id>/nodes/workflows?name=Graph.Dell.Racadm.Update.Firmware

{
  "options": {
          "defaults": {
                        "filePath": "xyz",
                        "serverUsername": "abc",
                        "serverPassword": "123",
                        "serverFilePath": "def"
           },
   "set-boot-pxe": {
                        "obmServiceName": "ipmi-obm-service"
                        },
   "reboot": {
                        "obmServiceName": "ipmi-obm-service"
   }
 }

In Band Management Settings¶

Get list of In Band Management settings that have been associated with nodes.

Get list of IBMs settings

GET /api/current/ibms

curl <server>/api/current/ibms

Get list of IBMs schemas showing required properties to create an IBM

GET /api/current/ibms/definitions

curl <server>/api/current/ibms/definitions

Create or update a single IBM service and associate it with a node

PUT /api/current/ibms

curl -X PUT -H "Content-Type: application/json" -d '{ "nodeId": <node id>, "service": "snmp-ibm-service", "config": { "community": "public", "host": "<host ip>" } }' /api/current/ibms

Example output of PUT

{
  "id": "591c569c087752c67428e4b3",
  "node": "/api/2.0/nodes/590cbcbf29ba9e40471c9f3c",
  "service": "snmp-ibm-service",
  "config": {
    "host": "172.31.128.2"
  }
}

Get a specific IBM setting

GET /api/current/ibms/<id>

curl <server>/api/current/ibms/<id>

PATCH an IBM setting

PATCH /api/current/ibms/<id>

curl -X PUT -H "Content-Type: application/json" -d '{ "nodeId": <node id>, "service": "snmp-ibm-service", "config": { "community": "public", "host": "<host ip>" } }' /api/current/ibms/<id>

Delete an IBM setting

DELETE /api/current/ibms/<id>

curl -X DELETE <server>/api/current/ibms/<id>

Node Tags¶

Add a tag to a node

PATCH /api/current/nodes/<id>/tags

curl -H "Content-Type: application/json" -X PATCH -d '{ "tags": [<list of tags>]}' <server>/api/current/nodes/<id>/tags

List tags for a node

GET /api/current/nodes/<id>/tags

curl <server>/api/current/nodes/<id>/tags

Delete a tag from a node

DELETE /api/current/nodes/<id>/tags/<tagname>

curl -X DELETE <server>/api/current/nodes/<id>/tags/<tagname>

Node Relations¶

List relations for a node

GET <server>/api/current/nodes/<id>/relations

curl <server>/api/current/nodes/<id>/relations

Sample response:

[
  {
      "relationType": "contains",
      "targets": [
            "57c0d980851053795fdc7bcf",
            "57c0d6bd851053795fdc7bc4"
          ]
    }
]

Add relations to a node

PUT <server>/api/current/nodes/<id>/relations

curl -H "Content-Type: application/json" -X PUT -d '{ <relationType>: [<list of targets>]}' <server>/api/2.0/nodes/<id>/relations

Sample request body:

{
    "contains":  ["57c0d980851053795fdc7bcf", "57c0d6bd851053795fdc7bc4"]
}

Sample response body:

[
  {
      "autoDiscover": false,
      "createdAt": "2016-08-30T18:39:57.819Z",
      "name": "demoRack",
      "relations": [
            {
                "relationType": "contains",
                "targets": [
                    "57c0d980851053795fdc7bcf",
                    "57c0d6bd851053795fdc7bc4"
                ]
            }
          ],
      "tags": [],
      "type": "rack",
      "updatedAt": "2016-08-30T21:07:11.717Z",
      "id": "57c5d2fd64bda4e679146530"
    },
  {
      "autoDiscover": false,
      "createdAt": "2016-08-27T00:06:24.784Z",
      "identifiers": [
            "08:00:27:10:1f:25"
          ],
      "name": "08:00:27:10:1f:25",
      "relations": [
            {
                "relationType": "containedBy",
                "targets": [
                    "57c5d2fd64bda4e679146530"
                ]
            }
          ],
      "sku": null,
      "tags": [],
      "type": "compute",
      "updatedAt": "2016-08-30T21:07:11.729Z",
      "id": "57c0d980851053795fdc7bcf"
    },
  {
      "autoDiscover": false,
      "createdAt": "2016-08-26T23:54:37.249Z",
      "identifiers": [
            "08:00:27:44:97:79"
          ],
      "name": "08:00:27:44:97:79",
      "relations": [
            {
                "relationType": "containedBy",
                "targets": [
                    "57c5d2fd64bda4e679146530"
                ]
            }
          ],
      "sku": null,
      "tags": [],
      "type": "compute",
      "updatedAt": "2016-08-30T21:07:11.724Z",
      "id": "57c0d6bd851053795fdc7bc4"
    }
]

Remove Relations from a node

DELETE <server>/api/current/nodes/<id>/relations

curl -H "Content-Type: application/json" -X DELETE -d '{ <relationType>: [<list of targets>]}' <server>/api/current/nodes/<id>/relations

Sample request body:

{
    "contains":  ["57c0d980851053795fdc7bcf", "57c0d6bd851053795fdc7bc4"]
}

Sample response body:

[
  {
      "autoDiscover": false,
      "createdAt": "2016-08-30T18:39:57.819Z",
      "name": "demoRack",
      "relations": [],
      "tags": [],
      "type": "rack",
      "updatedAt": "2016-08-30T21:14:11.553Z",
      "id": "57c5d2fd64bda4e679146530"
    },
  {
      "autoDiscover": false,
      "createdAt": "2016-08-27T00:06:24.784Z",
      "identifiers": [
            "08:00:27:10:1f:25"
          ],
      "name": "08:00:27:10:1f:25",
      "relations": [],
      "sku": null,
      "tags": [],
      "type": "compute",
      "updatedAt": "2016-08-30T21:14:11.566Z",
      "id": "57c0d980851053795fdc7bcf"
    },
  {
      "autoDiscover": false,
      "createdAt": "2016-08-26T23:54:37.249Z",
      "identifiers": [
            "08:00:27:44:97:79"
          ],
      "name": "08:00:27:44:97:79",
      "relations": [],
      "sku": null,
      "tags": [],
      "type": "compute",
      "updatedAt": "2016-08-30T21:14:11.559Z",
      "id": "57c0d6bd851053795fdc7bc4"
    }
]