Task Service¶
1. Introduction¶
Task Service allows execution of batch programs in a distributed, scalable, durable and highly available manner. In TCS Connected Universe Platform (TCUP), Task Service facilitates the processing of many concurrent asynchronous tasks/ jobs. Typical processing involves various algorithms and model execution to generate necessary analytical insight.
TCUP Task Service is designed to work in Master-Worker architecture pattern comprising of multiple distributed nodes and can elastically scale up/ down on need basis. The tasks/ jobs are maintained in a queue and gets executed on a cluster of nodes.
Master: Task Service consists of one or more master node(s). The master node is responsible for accepting all user/ API requests related to the task and dispatches the same to the worker nodes for execution. Optionally master can also be configured to work as one of the worker node to execute tasks.
Worker: The tasks/ jobs are actually executed on worker nodes. Worker consumes the task from task queue asynchronously and processes them. Each task/job gets packaged as container which runs independently to execute the underlying task. The container stays live till the task finishes its execution and eventually gets destroyed once the task finishes. The task sends/ stores all necessary output file in shared file system before getting destroyed.
Architecture: Task Service is implemented with a distributed architecture, scaled over multiple systems. Task Service API creates and manages various tasks as created by user. The metadata and information regarding task execution is stored in Redis DB as primary storage. Task processors maintained by the cluster are connected to each other and to the web server via same Redis endpoint. Task processors execute tasks in docker containers created from images based on user-defined runtime environment. The files required for task execution are shared across all task processors over NFS, which are mounted onto the container during container creation. Task lifecycle events are captured using docker event stream and are propagated to user interface at real-time.
Supported Languages: TCUP Task Service offers runtime to execute batch program written in various programming languages such as Java, C, Python, Spark, NodeJS etc. The developer doesn’t need to set-up/ configure the runtime to deploy and execute task rather they only need to focus on writing the business logic.
1.1 Intended Audience¶
The intended audience of this document is anyone who wants to have an overview of TCUP Task Service. After going through this document, the user will understand the capability of TCUP Task Service in IoT platform.
2. Key Concepts¶
In order to use the Task Service, a user needs to understand some of the basic concepts and building blocks of Task Service. Please refer to the following section for the concepts:
2.1 Project¶
A Project consists of a group of related tasks. Tasks must be grouped under a project. The prerequisite to create a task in Task Service is to create a project by specifying a project name.
2.2 Tasks¶
A task is an executable package of a program. It goes through the following lifecycle phases as listed below:
Create Task –Task is created under a project by specifying a task name. Once user creates the task ,the status become Created and task name and task ID are assigned to it.
Deploy Task – Once the task is created, user has the option to deploy the task. The following parameters are required to deploy the task:
Executable file – This is a program file which will be actually executed while running the task. This is a mandatory parameter to deploy a task.
Input file – Any data file required by the executable as input.
Dependent file – Any dependent file required by the executable as input.
Miscellaneous – Any other file if required by the executable.
Run time arguments – Any arguments required by the executable at the run time.
Run time environment – User mentions the runtime environment like Java, C etc. for task execution depending upon the executable file. This is a mandatory parameter to deploy a task.
Once deployed the task status become Deployed. User also has the option to Undeploy and Redeploy a deployed task.
Start Task – Once deployed, the task is ready to run. User can start the task providing the task ID. When started the task status becomes Running and after successful completion the status becomes Finished. While running if there is any error, its status becomes Error.
Stop, Pause, Resume Task – User has the option to stop, pause a running task and again resume a task when required.
Apart from standard task creation and deployment there is support for different task scheduling options as listed below:
2.3 Delayed Tasks¶
This is similar to a standard task and the only difference is while creating the task user has to specify a time in millisecond by which the task will be delayed and get executed in future time.
2.4 Recurrent Tasks¶
This task is also similar to a standard task and the difference is while creating the task user has the option to specify a frequency at which the task will be executed repeatedly (like a cron job).
3. Functional Capabilities¶
The Task Service provides the following functional capabilities:
Allows a user to create and register a project which holds similar kind of tasks. Once created, system will return a unique ID which is needed to create tasks in the future.
Allows a user to create a task with a meaningful name under a particular project. Upon successful registration, a unique ID for the task will be returned to user.
Allows user to upload executable programs in supported languages, input data files, dependent data file using exposed API or with help of user interface. Task Service will package all content into a runnable container and make it available for execution.
Allows to run deployed task on a distributed cluster. Initially task is being queued and dispatched later to the available machines in the cluster. Task status gets updated automatically if user subscribes to particular task event channel.
Supports unix cron pattern for scheduling. A deployed task can be scheduled based on a future time or it can be run with defined number of cycle with time or in a fixed time interval.
Allows task to be paused or deleted permanently.
After successful run, task results can be fetched through user interface or using API. If task fails to run, an error message will also be generated and displayed.
A Task can be delayed (i.e run at a future time instance) as per customer’s needs.
A Task can also be run recurrently in a cron-like interval.
Pipeline of task execution can be created by linking one task’s output to another task’s input.
4. Purpose/Usage¶
Task Service is a fully managed TCUP service that enables developers and data scientists to quickly and easily run any supported executables at any scale including machine learning models. TCUP Task Service reduces all the barriers that developers typically face for data analytics in production system. This service manages all the underlying infrastructure for you and provides a simple web API and interactive user interface to manage full life cycle.
Some of the common usage patterns are as follows:
Run Analytics using Task Services consuming data stored in Sensor Observation Service.
If messages from sensors/ devices are available in bulk as a file, a data ingestion task can be created to process these files in parallel and insert data to data storage or Sensor Observation Service.
Task Service can be used to trigger an external event if required after completion of task.
5. Examples¶
The following are examples of the usage of Task Service:
Use Case1¶
Imagine a use-case where weather data is captured and stored in either TCUP Sensor Observation Service or in a file. Our job is to find basic analytics like the day’s low temperature, month’s average rainfall or more advanced work like predicting temperature for next month or relation between temperature and humidity.
To find out, we could write an analytics program written with Python or any supported language. The analytics program can be submitted to Task Service for execution. The task will do necessary computation on the provided data and generate an output file. The output file can be later accessed via Task service API or via the Task service web user interface. This output can be used to build a report or used in some other third party application.
Use Case2¶
Consider an example of an engine scheduled maintenance prediction system. The engine has an Electronic Control Unit (ECU) attached and this ECU is connected to a gateway which stores data in file format in a periodic manner in a file system. Each file contains various engine specific parameters like speed, temperature, pressure etc. Our objective is to forecast maintenance activity for engines.
This could be achieved by applying machine learning techniques on above dataset. Machine learning algorithms requires many phases of sequential execution. Data preparation, feature extraction, building model, train model, deploy model and finally predict or recommendation are few basic phases which needs to be connected to each other to do the job. We could consider each phase as separate task and connect each task’s output to next task’s input.
6. Reference Document¶
For more details about this service please refer the following documents
API Guide