TCUP Edge¶
1. Introduction¶
TCUP Edge is an IoT platform meant to run on resource constrained (low footprint) devices for addressing the needs of the edge tier. This is installed on IoT gateway devices and provides the ability of acquiring data, pre-processing and filtering, local data storage, edge analytics and cloud synchronization. TCUP Edge helps reduce dependency on the internet / WAN connection and enables the system to work without cloud connectivity. It helps in reducing data traffic to cloud and associated latencies since the processing algorithms can be run locally.
TCUP Edge can be installed in any flavors of Linux OS and supports both containerized and non-containerized deployment. TCUP Edge can be installed on a device with memory footprint (RAM) as low as 250 MB.
It offers out of the box connectors for commonly used protocols like MQTT, BLE, COAP, Zig Bee, CAN Bus, MOD Bus, OPCUA etc. It comes with an administration portal that runs on the device, which provides management console along with user interfaces for various monitoring and configuration tasks.
1.1 Intended Audience¶
The intended audience of this document is anyone who wants to have an overview of TCUP Edge Service. After going through this document, the user will understand the capability of TCUP Edge Service as an IoT platform.
2. Key Concepts¶
In order to use TCUP Edge Service, a user needs to understand some of the basic concepts of the service. Please refer to the following section:
2.1 SOS Lite¶
SOS Lite is data management service used for storage and retrieval of sensor time series data at the edge. It provides a RESTful API interface to access the local data storage. The following are the APIs for SOS Lite:
Sensor - A sensor is a device that detects and responds to some type of input from the physical environment. In SOSLite, a sensor can be created, searched and its observation can be deleted. There are three APIs dedicated to it i.e. registerSensor(POST), getSensorByName(GET), and getSensorBySynchronizationAttribute(GET).
Feature - Feature is usually associated with sensor, it defines an attribute which is measured or analysed. For example, a sensor can have temperature as a feature which it measures. A sensor can have multiple features. There are three APIs dedicated to feature i.e. registerFeature(POST), getfeatureByName(GET ) and getFeatureBySynchronizationAttribute(GET).
Observation - With the help of this component, observations for a sensor with its respective features can be started, obtained and deleted. There are six APIs dedicated to it i.e. registerObservation (POST), latestObservationsBySensorName (GET), latestObservationsByDate (GET), deleteObservationsById (DELETE), deleteObservationsByBeforeDate (DELETE), getObservationBySynchronizationAttribute (GET).
Generic API - This API (PUT) is used for updating flag of recorded observation table.
2.2 Cloud Data Synchronizer (CDS)¶
It is used for synchronization of data stored locally at edge to cloud back-end. Data can be posted to TCUP cloud or other cloud platforms. It has support for converting all the stored data to ‘CSV’ or ‘Parquet’ file format before publishing it to cloud. It can also be configured as a file listener which can consume any incoming files and synchronize the same with cloud platform.
2.3 SOS Lite Data Truncator (SDT)¶
This service is used for truncation of data in SOS Lite. Data which has already been synchronized to the cloud can be truncated from local storage. This is used to conserve local disk space. For example, if a user configures the time parameter as 1 day, then all the data till midnight of the day before yesterday will be uploaded to TCUP cloud. If user wants to upload data up to present time, then the aforesaid parameter value needs to be set as 0 day. Once the data is synchronized successfully it updates the flag of synchronized observations to ‘true’ in SOS Lite DB.
Note
CDS and SDT run as utility programs not as services.
They don’t have any API exposure.
Both of them have to be invoked via Cron.
Once the job is done the process is terminated.
The config file for CDS and SDT is different.
They both generate a log file and status file where they store the operation status.
Cron - The software utility cron is a time-based job scheduler in Unix-like computer operating systems. Cron driven by a crontab (cron table) file, is a daemon, a configuration file that specifies shell commands to run periodically on a given schedule. The crontab command opens the crontab for editing and lets the user add, remove or modify scheduled tasks. Both CDS and SDT, are scheduled in separate crontab files. While creating a container the crontab files for CDS and SDT is mounted on the container using docker volume or bind mount command, so any changes made on host filesystem are reflected on docker filesystem.
2.4 MQTT Gateway¶
An MQTT (Message Queuing Telemetry Transport) gateway is provided for receiving data from sensors that transmit data over MQTT protocol.
2.5 API Gateway¶
API Gateway is an interface to all the TCUP Edge components. It is responsible for establishing secure communication channel with external clients trying to communicate with edge components. All communication takes place over HTTP(s) and MQTT(s) protocols between different modules of TCUP Edge, API gateway enables security by supporting all these communications over SSL layer.
2.6 Log Synchronizer¶
Log Synchronizer is used for aggregation of log files generated across different components and store it in a compressed format. It is also responsible for sending these compressed logs to the cloud for long term storage and analysis.
2.7 Log Truncator¶
Log Truncator is used for deletion of compressed logs already synchronized with cloud.
2.8 Stream Processing over Rule-Based Computing Engine (SPRUCE)¶
This service is used for executing real time filters and analytics on observation data. It can be configured through a set of rules and is also capable of executing inbuilt actions and custom models on batched real time data.
2.9 Portal¶
It is a graphical user interface from where users can get better insight of edge platform behavior. It also provides control features where tuning and configuration of platform can be done based on requirement.
2.10 Algorithm Execution Engine (ANEX)¶
This service provides a framework for executing machine learning models. Custom models and algorithms developed is downloaded to TCUP Edge and executed using the ANEX service.
2.11 OTA Agent¶
This agent offers RESTful APIs to facilitate different functions such as resource query, resource update, sending update for a device etc.
3. Functional Capabilities¶
Assuming that all the components of TCUP Edge is installed through TCUP Edge installer, there are several functionalities on the edge which can be observed as mentioned below:
Data Acquisition (DM agent, custom agent) -
TCUP Edge provides various connectors which can connect to different devices over various protocols like MQTT, HTTP(S), OPCUA, Mod bus, CAN bus, RS 232, BLE etc. These connectors collect the data from equipment or devices and send it to local MQTT channel. Once data is received over MQTT channel different types of real time stream processing implementations can be carried out on top of it based on pre-configured rules and it can be routed to various end points.
Data Synchronization & Periodic Cleanup -
In most of the cases there are memory related issues when it comes to storage of recorded observations on the edge. Keeping these kinds of requirements in mind there is a service called Cloud Data Synchronizer (CDS) which provides various features to synchronize recorded data to the cloud in different formats. Data can be posted to any cloud end point including TCUP for permanent storage of these records. Data can be posted in the form of JSON or in a csv/ parquet file format to the cloud from the edge. It is also capable of enabling a listener to a particular folder to receive any files that has to be instantly posted to the cloud based on business requirement. Once the files are synchronized, it can be deleted from the file system or can be set in a archive which can be periodically deleted.
There is an event manager that keeps a record of all the internal activities happening on the edge box. Once the synchronization of record is successfully carried out these records can be periodically deleted from the file system through SOS Lite Data Truncator (SDT) which is again deployed as part of the edge platform. Both CDS & SDT can be configured as a periodic background process or can be executed on demand.
Monitoring -
TCUP Edge components can be deployed as both containers and in non-container mode. However, the monitoring of these components is vital to ensure smooth functioning of the program. There is a monitoring agent running in the background which captures the real time health status of not just the edge box but also of all the service components deployed on the edge box. These statistics can be viewed through user portal. Also various rules are configured as part of inbuilt monitoring system which raises alert in case if the system component breaches certain threshold. These alerts are classified under the category of system alerts as a part of the user portal.
User Portal -
It provides an interface to get the insight of various activities happening in a gateway device. It reflects the real time health status of all the components hosted as a part of the edge offering. A user can define various filtering and analytics rule through this portal. Different kinds of system and application alerts and notifications can be viewed. Control settings are provided to turn on or turn off any component which is deployed as part of the edge ecosystem. It provides direct visibility to various devices and sensors posting data to the gateway with real time charts. Also various background processes and their execution status can be monitored and configured.
Over The Air(OTA) Software Upgrade -
OTA is one of the most widely used feature of TCUP Edge. It enables up-gradation of existing services and deployment of custom applications onto the edge box directly from the cloud. It can also be used to send remote instruction to the devices or execute commands on the device.
Log Aggregation & Truncation -
Logs generated from all the components is aggregated locally and kept in a local file system. There is a file compression technique enabled by default which compresses the logs generated from every component and sends it on a regular basis to the cloud. These logs are often handy for debugging purpose. Once these logs are synchronized to the cloud, it is again truncated from the devices to release the storage space. These jobs of log synchronization and truncation can be controlled and administered from the portal.
Data Management -
Local data repository is available out of the box as a default offering. All the data ingested in gateway can reside locally through a service component known as Sensor Observation Service(SOS Lite). It is a light weight version of TCUP SOS. Internally it saves all the ingested records in a light weight high performance time series database. This service is available as a set of REST APIs. These records can be synchronized with the cloud. It can also be queried from any application through APIs.
Local(Edge) Filtering & Analytics -
TCUP Edge provides capabilities for filtering and analytics on individual stream of real time data as well as batch oriented analysis on a window created on real time streaming data. It has out of the box filtering and analytics capabilities which can be configured as part of the rule on any sensor’s observation out of many devices posting data onto the edge box. Few action types are already provided as part of platform offering and can be configured as part of the rule. These actions will be invoked based on rule settings.
TCUP Edge provides a provision to deploy and execute custom build models on observation records. Any custom developed model can be deployed to the edge by leveraging Over The Air (OTA) software upgrade service and through user portal. Configure rule option can be configured to create a pipeline where a batch of records can be passed as an input to the deployed model and output can be collected over MQTT channel. Similar rules can be configured to raise an alert or notification based on outcome of the model. Since these analytics are happening at almost real time and is closer to the data source, there is a significant reduction of latency involved in posting these batch of records to the cloud and wait for the outcome of model executed on these records to trigger any action.
9. Calculation Engine -
Calculation engine on the Edge is used to save time, increase efficiency, and allow significant level of process control. It is a real time calculation layer that runs in parallel with the process/experiment. The calculation layer uses the inputs from the process and runs user defined computations (y=f(x)) where x is the input tag series and y, is the output series of the calculation. The calculation output along with the input are monitored by the users to calculate/determine new offsets for effective process control.
4. Purpose/Usage¶
TCUP Edge is a full grown IoT platform which can be easily configured based on specific solution requirement. It is out of the box and possess most of the commonly used IoT capabilities which can be leveraged to float any IoT application based on existing infrastructure. It provides the ability for data acquisition, cleaning, pre-processing and filtering, local data storage, edge analytics, data synchronization to different cloud platforms, real time monitoring, alert and notification service, over the air software upgrade, user portal etc.
It helps to reduce dependency on internet/ WAN connection and enables the system to work in offline mode or even with intermittent network connectivity. It can work with both IoT enabled machines as well as legacy devices. It helps in reducing data traffic to cloud and in reducing latencies associated with control loops since the control algorithms can be run locally.
Any full-fledged application can be developed on it with better control and minimal development effort.
5. Examples¶
Consider an example of an Engine Monitoring System in a power plant. The Engine has an Electronic Control Unit (ECU) attached and this ECU is connected to a gateway over the CAN Bus protocol. This gateway is a device where TCUP Edge is installed. All the data coming via ECU is posted to the TCUP Edge.
TCUP Edge is configured to read all these observations and perform filtering and analytics on the ingested streaming data on a real time basis. Again based on the data received it is responsible to trigger the alert or notification and post selective data to cloud.
All incoming data is stored in a local database storage using Sensor Observation Service (SOS Lite). All the stored data is synchronized with the TCUP cloud through Cloud Data Synchronizer (CDS) and the synchronized records are then truncated on a regular basis through SOS Lite Data Truncator (SDT).
Once the rules are set and data is ingested, operations are automatically performed on the incoming data and action rules are triggered accordingly at Edge.
Additionally, rules are configured to invoke algorithm/ models on incoming data stream using the batch analytics framework. A custom developed model is pre-trained on the cloud, based on historical/ static dataset and is transmitted over to the edge box using Over The Air (OTA) software upgrade service along with required dependency files. A pre-configured rule invokes the model and passes data streams to the model to generate insights and decisions.
In case of any health issues of the ECU an alert is raised which is visible on the portal. Also actuation commands were send to the ECU over CAN Bus protocol from the cloud using Device Management (DM) service through Edge.
Various charts were drawn and dashboards wereconfigured based on historical data on the cloud. Also the meta details and asset hierarchy were visible on Asset Service portal on the cloud.
In a nutshell, Edge Services is used to:
Address low latency requirements enabling edge/ local data processing and analytics and real time decision making
Data buffering in case of loss of connectivity/ intermittent connectivity.
Reduce bandwidth cost by selectively posting data to cloud based on rules
Work with different legacy and modern devices over various protocols.
6. Reference Document¶
For more details about this service please refer to the following documents:
API Guide