Telemetry Providers
Technical Integration Guide
This guide is intended for use by those who have telemetry devices that need to send data to the Data Interchange. Before proceeding with integration, your organization should have received an access token and connection details from the Telemetry Provider onboarding process, as described above.
As a short overview of the process, a single HTTP POST request should be sent to the Data Interchange's publish API for each raw telemetry module report. For avoidance of doubt, be aware that this document uses the terms report, message, and event interchangeably to mean a package of data that is transmitted via a single HTTP POST to the publish API of the Data Interchange. The payload of the HTTP request should contain the data from the report. The request headers must include a small number of required headers, plus any additional headers (see: Message Headers) that can be populated from the source data system.
A data dictionary describing the nature and structure of the data being transmitted (e.g. sensor locations, expected values, average/min/max, sampling intervals) should be created and published through the Data Interchange every 24 hours. This enables Telemetry Consumers to receive a current Data Dictionary early in the integration cycle, and provides a simple mechanism to receive updates as the Data Dictionary evolves. Developing a Data Dictionary will allow Telemetry Consumers to understand and correctly process data that flows across the Data Interchange. Further guidance – including an example data dictionary – are available from New Horizons or at Data Dictionary Guidelines.
Publishing Events to the Data Interchange
Each data payload should be sent to the Data Interchange via a POST request to the data publishing endpoints (details below). There is a set of required metadata headers that must be included with each POST, which will be used to appropriately route the report to the Interchange data consumer specified by the Data Owner. In addition to the required metadata, a list of strongly recommended but optional metadata is detailed below.
Each HTTP POST sent to the Data Interchange is composed of three important sections:
- Endpoint
- Headers
- Payload
Endpoint
There are two API endpoints used for sending data into the Data Interchange. These endpoints are identifiedbelow. Both endpoints enforce the same requirements regarding metadata headers.
| URL | Notes |
|---|---|
| https://ix-publish.2to8.cc/publish/data | This endpoint is intended for streaming data and does not support file upload. |
| https://ix-publish.2to8.cc/publish/file | Thendpoint supports uploading of one or more files via multipart/form-data. |
Headers
Because there are many telemetry modules reporting data to the Data Interchange, there needs to be a mechanism for the Data Interchange to determine where to route each data message. The required headers satisfy this need. At a minimum, the ce-source, dx-owner, and dx-token headers must be included in the HTTP request. These headers enable the Interchange to identify the Telemetry Provider that sent the data and the Data Owner that owns that data. In addition, to support downstream consumers and conform to the CloudEvents specification, the ce-id, ce-type, and ce-specversion headers are also required.
As noted previously, it is strongly recommended to include as many additional headers from the complete list (see: Message Headers) that can be populated from the source data system. These headers assist Data Owners and downstream data-processing solutions (i.e., reporting dashboards) to understand and correctly display data contained in the the reports. For instance, the header dx-tdl-serial allows a Telemetry Provider to identify the data logger from which the data originated.
Payload
The POST's payload contains the raw data of the event. This data is the information provided in the telemetry module's report. The Data Interchange does not enforce rigid constraints on the request payload. Request bodies are forwarded as-is from Telemetry Providers to the Telemetry Consumers.
Generally, the payload contains batches of logs from data loggers being used to monitor cold chain equipment. The data can be used by reporting tools to provide reports about the object being monitored. For instance, the payload is likely to contain the following data points collected by a cold chain telemetry module: vaccine chamber temperature, ambient temperature, door open count, power state, and other relevant information. It is also acceptable for the payload to be non-telemetry data such as images, documents, and maintenance reports pertaining to a telemetry device. The Content-Type header ce-type and ce-dataschema headers should be updated accordingly to indicate the content of the payload.
The current max size for a payload is 2 MB. Ideally, report payloads will be kept under 1MB.
Required Metadata (Headers)
At least some metadata must be included in the headers of each published report. Each HTTP POST must include the following headers:
| Name | Description |
|---|---|
| ce-id string | The unique identifier for this data report. Example : 831-002::2020-09-18T12:01:38.376Z |
| ce-specversion string | The CloudEvent spec version (https://cloudevents.io/) Example : 1.0 (this is the only supported value at present) |
| ce-type string | The type or classification of the transmitted data report. Example : org.nhgh.varo.report.v1 |
| ce-source string($uri) | The source of the event. The value of this field may be any valid URI. Example : /widget-maker/telemetry/v1; https://telemetry.widget-maker.org/v1 |
| dx-token string | An access token, provided by the Data Interchange, that permits publication of data to the Interchange. Example : 675F928zkT993xjfT1ZL0Aprn5 |
| dx-owner string{$uri} | URI of the Data Owner, to which the data transmission should be routed. Example : https://health.gov.ng |
The values for ce-source, ce-specversion, dx-owner, and dx-token should have been provided by the Data Interchange Administrator during the onboarding process. The required ce-id and ce-type should try to follow the guidelines in the notes above. If you are unclear on the correct values, please contact Data Interchange Administrator (see Questions and Support section, later in this document) .
Strongly Recommended Metadata
In addition to the required metadata headers, there is additional metadata that is strongly recommended to be sent with each published event, if such data is available. Although the Data Interchange does not enforce hard requirements on the inclusion of this data, Telemetry Consumers have a reasonable expectation that such metadata is included whenever it is available. This metadata can be used to provide context to the reporting tool/processor consuming the data from the Data Interchange. The Data Interchange does not use this additional metadata for routing, it simply passes it through to the Telemetry Consumer(s). For complete details on metadata Headers, see Message Headers.
Batching Reports
If possible, it is recommended that reports for a given logger be aggregated and sent in a single, batched request (for example, one batched delivery every hour or 4 hours). A batch transmission must not exceed 2MB in size. If a batch of 1,000 samples is transmitted, the ce-time value should be the same as or after the time of the most recent sample in the batch. Batched requests should only be sent for data from a single, individual telemetry module.
When batching data it is recommend each sample in the batch have its own UTC timestamp, reflecting the time when the data was measured by the telemetry module. Additionally, if the sample rate is variable a monotonically increasing sample count number can be used to help identify data gaps.
Example Post
Below is an example POST request to publish data, which contains a single report. This request includes the required headers, plus some optional metadata headers:
curl -X POST \
https://ix-publish.2to8.cc/publish/data \
-H 'Content-Type: application/json' \
-H 'ce-id: {{ Unique Event ID }}' \
-H 'ce-source: {{ Telemetry Provider Source URI }}' \
-H 'ce-time: {{ ISO Formatted Timestamp }}' \
-H 'dx-country: {{ Country Alpha 2 Code }}' \
-H 'dx-owner: {{ Data Owner URI }}' \
-H 'dx-facility-name: {{ Facility Name }}' \
-H 'dx-fridge-assigned-id: {{ Fridge ID}}' \
-H 'dx-fridge-manufacturer: {{ Fridge Manufacturer}}' \
-H 'dx-fridge-model: {{ Fridge Model }}' \
-H 'dx-location-latitude: {{ Latitude Value }}' \
-H 'dx-location-longitude: {{ Longitude Value }}' \
-H 'dx-tdl-manufacturer: {{ Logger Manufacturer }}' \
-H 'dx-tdl-model: {{ Logger Model }}' \
-H 'dx-tdl-serial: {{ Logger Serial }}' \
-d '[{
"hamdaq": 88.4000015258789,
"susb": 4,
"fridgeId": 2887014069330542759,
"tcpdisctl": 26.5,
"ewdc": 4.96999979019165,
"tvctopctl": 3.190000057220459,
"icps": 0,
"measuredDtm": "2020-09-18T12:02:54.892Z"
}]'
The values wrapped in {{ }} above should be replaced with valid values from the given telemetry module and report, where available. For instance, if the fridge manufacturer is known to be “Widget Maker” then the following header:
-H 'dx-fridge-manufacturer: {{ Manufacturer Name }}'
Should look like this:
-H 'dx-fridge-manufacturer: Widget Maker'
The request payload (content after the -d parameter) should be replaced with a batch of data from the telemetry module. The keys and values do not need to match those in this example. The payload does not need to be JSON. Binary and non-JSON payloads may also be sent, but the Content-type header should be updated accordingly.
Data Validation
To verify that data is successfully being received and routed by the Data Interchange, Telemetry Provides can login to the Data Interchange and navigate to the Metrics section. The Metrics section provides charts and information to help Telemetry Providers understand the flows of data through the Data Interchange. You may also contact the New Horizons team directly to verify successful receipt of data.
Questions and Support
For additional information or clarifications on the above, and for confirmation of successful data transmission to the Interchange, feel free to email New Horizons directly at datatools@nhgh.org