Timeseries Metric Service

1. Introduction

Timeseries Metric Service (TMS) enables the timeseries data management capabilities in TCS Connected Universe Platform (TCUP). Any measurement that is repeatedly generated from any source over a prolonged duration of time, is called timeseries data for example, the heart rate (metric) of a person collected though wearable devices (source) over an hour. Timeseries data management is also a fundamental capability of any industrial IoT platforms like TCUP.

TMS is a TCUP microservice that helps in acquisition, persistence, filtering, and retrieval of timeseries data. Over the years in multiple use cases of TCUP, different data ingestion workload patterns have been observed. TMS being a relatively new microservice enjoys the benefits of cutting-edge technologies to support high-frequency insertions, batch insertions, real-time streaming inputs and so on. The performance is much better than existing similar APIs like the insert observation API in Sensor Observation Service (SOS). The data stored is also efficiently optimized to reduce storage requirement therefore reducing cost for the overall solution.

TMS is developed ground-up with different concepts than TCUP SOS. The APIs here are simpler and more flexible. TMS is more scalable to load and robust in system failures. TMS is a better alternative to SOS for insertion and retrieval of sensor data. Also, it is functional in conjunction with SOS, where sensor metadata (feature, sensor, observed properties) is kept in SOS, but instead of keeping observation data in SOS, it could be stored in TMS. The schema and interfaces of the APIs of TMS and SOS are incompatible to each other, but using proper data modelling, the payload of SOS can be converted to TMS very easily by the client agents.

1.1 Intended Audience

The intended audience of this document is anyone who wants to build application architecture involving this service, consume APIs of this service or simply want to learn the capability of this service. This document is a pre-read of Timeseries Metric Service API Guide.

2. Key Concepts

The objective of TMS is to provide a performant way to insert timeseries data into TCUP, store it in storage efficient manner, provide search and filtering on historical data, as well as expose real-time feeds of timeseries events. The metadata associated to timeseries data is minimum as far as possible. The main entities of this microservice are as follows:

2.1 Metric

Metric is measurement of any parameter of a physical being which is normally generated or collected by any sensors or devices. For example, the temperature of an office floor captured over a period of time. In TMS, metric is not only restricted to have numeric value, but based on the requirements, it can have text, Boolean, timestamps, arrays or even a complex object as value. Even though there is scope for persisting any type of data as metric, the fundamental characteristics of timeseries metric is that it potentially changes the values over time. Having said that, in the earlier example, the office floor co-ordinates can be captured and persisted in TMS, but ideally it is not a metric. It will be a tag of a metric, which is another concept. Timeseries metric values must associate a timestamp along with it. If a timestamp is not provided while inserting a metric value, server time is attached. There is no temporal uniqueness requirement for timeseries metric, which means duplicate values may exist of a metric on a given timestamp. Apart from metric values, there can be additional metadata like timeseries source information, tags and position can be inserted along with metric, but all these are optional. Metric can be compared with the concept of observed property values of SOS. The only difference is that metric source could be more versatile than sensors in SOS. This might suit some use cases, where metric can be directly associated with an existing asset or device of TCUP.

Example of metrics:

  • temperature of a room recorded over a period of time.

  • speed of a vehicle from one location to another

Metric can be associated with location data in TCUP.

2.2 Source

Source is the identification of a physical or virtual entity which generates timeseries data. The timeseries data is measurement of one or more properties of some source. Sources can be sensors, devices, or any other assets (of TCUP) or even entities not recognized by TCUP. In TCUP, timeseries data may exist without source identification. This is not a usual scenario but an important design decision to cater plurality of use cases, such as - scenarios where source is not required in query or, applications dealing with data from single source or, dumping staged data for data preparation etc.

2.3 Tag

Tags are repetitive text value that may optionally associate with a timeseries metric. Tags are identified by a unique, numeric, non-updatable and non-deletable tagId. It is advised that TTS clients, download and cache the intended TagId before sending timeseries metrics to TTS.

For example:

2.4 Schema

The part that makes TMS flexible yet powerful, is the variety of user configurations available to make. Unlike other micro-services, this service allows users to configure timeseries storage schema so that it can be optimized both for performance and storage.

3. Functional Capabilities

The Timeseries Metric Service (TMS) has the following capabilities:

  • Create a user schema dynamically with workload hints (e.g. - number of sources, metrics per day, retention period etc.).

  • Enable location tracking, if required.

  • Define list of metrics with proper data types.

  • Get detail about the existing user schema, along with the metric definitions and the automatic data partition details.

  • Enable / disable metric data feed capability. Disconnect all feed subscribers in single API call.

  • Delete a metric definition if there is no existing value persisted for the metric.

  • Insert a metric value along with timestamp, source, and tags.

  • Upload a CSV file containing a large amount of such metric values.

  • Search metric values based on variety of filtering condition.

  • Provide simple aggregation, sampling, interpolation of metric values.

  • Download a CSV or compressed file containing metric data.

  • Get real time push notification for latest metric. Filter the notification criteria.

  • Create new metric tags, get the tags, and update existing tag values.

  • Create new source, get the detail, and update existing source type or description.

4. Purpose/Usage

TMS helps to acquire and persist sensor data. Some of the common usage patterns are as follows:

  • Transmission of measurements / event data from edge devices / gateways using TMS API calls, persistence of the observation data within TMS databases and retrieval of the same using TMS query APIs.

  • Uploading of data from files, such as CSV files, consisting of time-series data into TMS.

  • Generating visualization / graphs / charts using TMS query APIs or real-time feeds. Data from TMS query APIs is returned as JSON documents and can be easily rendered using any charting or graphing utility.

  • Analytics programs can pull data from TMS service using query APIs and generate and execute models. For example, analytical programs written in languages such as Python can call TMS query APIs to retrieve historical data.

5. Example

Consider an example of an Engine Monitoring System. The Engine has an Electronic Control Unit (ECU) attached and this ECU is connected to a gateway via the CANbus. The gateway is connected to TCUP via a wireless internet connection. The gateway runs an embedded Linux operating system. A data acquisition software is running on the gateway. This software uses Timeseries Metric Service to push data to TCUP cloud. The application is modelled in TCUP as follows: - User defines the Engine Speed, Engine Power Temperature, Pressure etc. as metrics in TMS. It is one-time activity. - The engine Id is registered as tag in TMS, and the system generated unique serial Id of this tag is cached at gateway for future data inserts. - The ECU is considered as the source (sensor) and all the values received from ECU for Engine Speed, Engine Power Temperature, Pressure etc., along with time of event, are sent to TMS. - The data acquisition software collects the data from the CANBus and pushes the data to TCUP using the TMS Insert Metric API. - An application on the cloud side uses TMS Get Metric API to retrieve data and generate reports as needed.

6. Reference Document

Please refer to the following documents for more details about this service:

  1. API Guide