Task Service

Introduction

Task Service is used for batch oriented data processing on historical sensor data stored and managed by external data services such as TCUP Sensor Observation System. IoT applications require the ability to process high volumes of data collected over a period of time and perform analysis on the data in batch mode. Typical processing involves machine learning algorithms for generating and executing analytical models.

Task Service enables a large number of batch programs running analytics workloads to be executed in parallel on large server clusters. Developers provide the path to the program and specify the path to the data. Using Task Service APIs, users can upload the program to be executed. Task Service APIs can be used to execute these programs written in C, Java, Python etc. Application developers have the option of managing the schedule and specifying the number of nodes that will execute the tasks.

Purpose of the Document

The following document describes how to work with task service using the portal. After going through this document, the users will able to perform the following activities:

  • Create a project

  • Create task under that project.

  • Deploy task with executable file, dependent files (in a .zip file), input files (in a .zip file) and runtime arguments.

  • Start task.

  • Pause/ resume, cancel and stop task if required.

  • Delete/ restart task.

  • Export existing task and import the exported task.

  • Delay tasks where the task can be deployed with a future execution time and

  • Recurrent tasks where the tasks can be deployed at recurrent intervals (daily, weekly, etc.) using Cron pattern and optional argument of recurrence limit (Value should be any positive number or 0 if task has to be run infinitely).

Reference Document

Please refer to the following documents to get more details on Task service

  • To understand the basic concepts of Task Service please refer to the Concept guide.

  • For API details please refer to the API guide.

Working on Task Service Using Portal

To navigate to Task Service portal, after signing in to TCUP, click on Services on left hand side menu and then click on Task .

../_images/portalNavigation.png

Project

A Project is group of related tasks. Tasks must be grouped under a project

Register Project

../_images/createProject1.png

A Project can be registered by clicking the register button present on the top right corner of the project page. This will open the registration form where the details have to be filled in.

../_images/createProject2.png

On successful registration, a unique projectId will be generated against the given project name.

../_images/createProject3.png

Search a Project

Project can be searched by just typing the project name or project ID on the top right search field.

../_images/searchProjectById.png

Delete Project

Any project can be deleted by clicking on the Delete button beside the project name.

../_images/deleteProject1.png

A popup appears asking to confirm whether to proceed with the deletion.

../_images/deleteProject2.png

Upon agreeing to proceed the project gets deleted successfully.

../_images/deleteProject3.png

Refresh Project List

Project list can be refreshed by clicking the refresh button as shown in the image below:

../_images/refreshProjectList1.png

Task

A task is an executable package of a program.

To enter the task page, the user has to click on the project name to fetch all the tasks created under a single project.

../_images/fetchTaskByProjectId1.png ../_images/fetchTaskByProjectId2.png

Register Task

A Task should be registered by clicking the register button present on the top-right corner of the task page.

../_images/createTask1.png

This will open the registration form where the task name has to be mentioned.

../_images/createTask2.png

1. Task Name

On successful registration, a unique taskId will be generated for the task.

../_images/createTask3.png

Refresh Task List

The icon shown in the image below allows to refresh the current task list that appears in the page.

../_images/refreshTaskList1.png

Search a Task

Tasks can be searched by typing the task name in the search box as shown in the image below.

../_images/searchTask1.PNG

Filter Task by Status

Tasks can be filtered by their status by just selecting the task status from the drop-down menu.

../_images/searchTaskByStatus1.PNG ../_images/searchTaskByStatus2.PNG

Deploy Task

On clicking the Action button against a task, Deploy button appears as shown in the image below:

../_images/deployTask1.png

On clicking on Deploy a form appears which is described below:

../_images/deployTask2.png

1. Select Task Run Environment

Here the run-time environment has to be selected in which the task will run. Python, Java, C, C++, Spark, Node-js are currently the supported formats.

2. Upload Executable File

Here the executable files have to be uploaded that will run according to the environment selected in the previous step.

3. Upload Input zip File

Here input files have to be uploaded in zip format that are needed for the executable file to run.

4. Do you want to add external location?

If we want output file of some other executed task or some local file to be the input of the current task then external location should be mentioned as ‘yes’ and the appropriate location type should be selected else the value should be ‘no’.

../_images/TaskExtLoc.png

For details of external file please refer Add External Input and Output Config to Task

5. Upload Dependent zip File

Here dependent files are uploaded in zip format that are required for the executable file to run.

Note

Input File is used to provide data to the batch program. For example, if a program needs to read data from 2 data files for processing, then those 2 data files should be provided as Input File. Dependent File is required for running the program. For example, if a Java program needs some jar file for running it, then that jar file should be provided as a dependent file.

6. Runtime Arguments

Run-time arguments can be provided if it is required in the uploaded program.

After filling up the required parameters, the ‘deploy’ button has to be clicked. A confirmation of the task being deployed will be provided as shown in the image below.

../_images/deployTask4.png

Start Task

../_images/startTask1.png

After deploying the necessary files, the START button has to be clicked to start the execution.

../_images/startTask1.1.png

Input files (.zip file for required data files) can be optionally uploaded and runtime arguments can be updated during the task execution.

../_images/startTask2.png

Once the execution of the program starts, the status is set to RUNNING.

../_images/startTask3.png

Once the execution of the program is completed, the status becomes FINISHED. The generated output files, start time, end time and the time taken to complete the execution can be viewed.

../_images/taskFinished.png

Stop Task

../_images/stopTask1.png

Any particular task can be stopped by clicking the STOP button.

../_images/stopTask2.png

The status of the task becomes ABORTED.

Pause Task

../_images/pauseTask1.PNG

Any particular task can be paused by clicking the PAUSE button.

../_images/pauseTask2.PNG

The status of the task becomes PAUSED.

Resume Task

../_images/resumeTask1.PNG

You can resume any particular task by clicking the RESUME button.

../_images/resumeTask2.PNG

The status of the task gets updated to RUNNING/QUEUED.

Refresh a Particular Task

../_images/refreshTask1.png

Any particular task can be refreshed by clicking the REFRESH button.

../_images/refreshTask2.png

Undeploy Task

../_images/undeployTask1.png

A task can be undeployed by clicking the UNDEPLOY button.

../_images/undeployTask2.png

The status of the task becomes UNDEPLOYED.

Delete Task

../_images/deleteTask3.png

Any particular task can be deleted by clicking the DELETE button.

../_images/deleteTask1.png

A popup appears to confirm whether one wants to proceed with the deletion.

../_images/deleteTask2.png

Upon confirmation the task gets deleted successfully.

External File Linking

It is used to add inputs and outputs to pre-configured destination.

../_images/external1.png

To add another task, the output file location type should be “task” and the corresponding project id, task id and output file to be mentioned as shown in the image below:

../_images/external2.png

To add a local file, the location type should be “local” and file name to be mentioned as shown in the image below:

../_images/external3.png

Delayed Task

Register Delayed Task

A delayed task can be registered by clicking the register button present on the top-right corner of the delayed task page as shown in the image below:

../_images/delayedTask1.png

A registration form will open where the following task details have to be filled in.

../_images/delayedTask1.9.png

1. Delayed Task Name

This is the name of the delayed task.

2. Execution Time

This is the execution date-time of the delayed task.

../_images/delayedTask2.png ../_images/delayedTask3.png

The date and time value have to be selected.

On successful registration, a unique taskId will be generated for delayed task.

../_images/delayedTask4.png

Refresh Delayed Task List

The icon shown in the image below allows to refresh the current task list that appears on the page.

../_images/delayedTaskSearchAll1.png

Search a Delayed Task

Tasks can be searched by typing the task’s name in the search bar as shown below:

../_images/delayedTaskSearch1.png

Filter Delayed Task by Status

Tasks can be filtered by selecting the task status from the drop-down menu.

../_images/delayedTaskFilterByStatus1.png

Deploy Delayed Task

../_images/delayedTaskDeploy0.png

On clicking the Deploy button a form appears as described in the image below:

../_images/delayedTaskDeploy1.png

1. Select Task Run Environment

Here run-time environment has to be selected in which the task will run. Python, Java, C, C++ ,Spark and Node-js are currently the supported formats.

2. Upload Executable File

Here executable files have to be uploaded that will run according to the environment that was selected in the previous step.

3. Upload Input zip File

Here input files have to be uploaded in zip format that are required for the executable file to run.

4. Upload Dependent zip File

Here dependent files are uploaded in zip format that are required for the executable file to run.

5. Runtime Arguments

Run-time arguments can be provided if it is required in the program uploaded in the previous step.

After filling up the required parameters, click the ‘deploy’ button. The task deployment confirmation will be displayed as shown in the image below:

../_images/delayedTaskDeploy3.png

Start Delayed Task

../_images/delayedTaskStart1.png

After deploying the necessary files, the execution can be started by clicking the START button.

../_images/delayedTaskStart1.1.png

While starting the execution there is an option to upload the input files(.zip file for required data files) and/or update the runtime arguments for task execution. Now the status of the program becomes QUEUED.

../_images/delayedTaskStart2.png

Once the execution of the program starts, the status becomes RUNNING.

../_images/startTask3.png

Once the execution of the program is completed, the status becomes FINISHED. After the execution is completed, the generated output files, start time, end time and the time taken to complete the execution can be viewed.

../_images/delayedTaskFinished.png

Stop Delayed Task

A delayed task can be stopped by clicking the STOP button.

../_images/delayedTaskStop1.png

Once stopped, the status of the delayed task becomes ABORTED.

../_images/delayedTaskStop2.png

Pause Delayed Task

A delayed task can be paused by clicking the PAUSE button.

../_images/delayedTaskPause1.png

The status of the delayed task becomes PAUSED as shown below.

../_images/delayedTaskPause2.png

Resume Delayed Task

A paused or delayed task can be resumed by clicking the RESUME button.

../_images/delayedTaskResume1.png

The status of the delayed task becomes RUNNING/QUEUED.

../_images/delayedTaskResume2.png

Refresh a Delayed Task

A delayed task can be refreshed by clicking the REFRESH button.

../_images/delayedTaskRefresh1.png

Reschedule Delayed Task

A delayed task can be rescheduled by clicking the RESCHEDULE button.

../_images/delayedTaskReschedule1.png

On clicking the reschedule button, the following form will appear:

../_images/delayedTaskReschedule1.1.png

1. Task Name

This is the name of the task.

2. Execution Time

This is the execution date-time of the task.

3. Input File(Optional)

This is a .zip file containing the data file for task execution.

4. Runtime Arguments(Optional)

This is the command-line arguments for task execution.

After entering the details, the ‘reschedule’ button has to be clicked on the form. The delayed task then gets rescheduled and queued successfully.

../_images/delayedTaskReschedule4.png

Undeploy Delayed Task

A delayed task can be undeployed by clicking the UNDEPLOY button.

../_images/delayedTaskUndeploy1.png

The status of the task becomes UNDEPLOYED.

../_images/delayedTaskUndeploy2.png

Delete Delayed Task

A delayed task can be deleted by clicking the DELETE button.

../_images/delayedTaskDelete1.png

A popup appears asking to confirm whether to proceed with the deletion.

../_images/delayedTaskDelete2.png

Upon confirming the delayed task gets deleted successfully.

../_images/delayedTaskDelete3.png

Note

  • Delayed Task is for one time execution.

  • Delayed date-time must be greater than the system date-time.

Recurrent Task

Register Recurrent Task

A recurrent task can be registered by clicking the Create New Recurrent Task button present on the top-right corner of recurrent task page.

../_images/recurrentTask1.png

This will open the registration form where the required details needs to be filled up.

../_images/recurrentTask2.png

1. Recurrent Task Name

This is the name of the task.

2. Cron Pattern

This allows to select the cron pattern of the recurrent task.

../_images/recurrentTask3.png

The recurrent interval of the task can be as follows:

a. Second

b. Minute

c. Hour

d. Day

e. Week

f. Month

g. Year

Two types of cron pattern can be set:

1. Tasks can be run at a fixed time of a day,every day, or on a fixed day and time of a week for every week, etc. For example, 7:30 AM every day, or 12:00 PM Monday, Tuesday, Wednesday every week, etc. 2. Tasks can be run at a regular interval of seconds, or minutes. For example, every 15 minutes or every 30 seconds, etc.

3. Counts

This allows to set the recurrence limit. If not set, i.e. if it is set at 0, the task will schedule and run infinitely.

On successful registration, a unique taskId will be generated for the recurrent task.

../_images/recurrentTask4.png

Refresh Recurrent Task List

The above button allows to refresh the current recurrent task list that appears on the page.

../_images/recurrentTaskRefreshAll1.png

Search a Recurrent Task

Recurrent tasks can be searched by just typing the recurrent task name.

../_images/recurrentTaskSearch1.png

Filter Recurrent Task by Status

Recurrent tasks can be filtered by status by selecting the task status from the drop-down menu.

../_images/recurrentTaskFilterByStatus1.png

Filter Recurrent Task by Schedule Status

Recurrent task can also be filtered by schedule status by selecting the schedule status from the drop-down menu as shown below:

../_images/recurrentTaskFilterByScheduleStatus1.png search_app

Deploy Recurrent Task

../_images/recurrentTaskDeploy0.png

On clicking the deploy button the form described below appears:

../_images/recurrentTaskDeploy1.png

1. Select Task Run Environment

Here the run-time environment has to be selected in which the task will run. Python, Java, C, C++, Spark and Node-js are currently the supported formats.

2. Upload Executable File

Here the executable files have to be uploaded that will run according to the environment that was selected in the previous step.

3. Upload Input zip file

Here the input files have to be uploaded in zip format that are required for the executable file to run.

4. Upload dependent zip file

Here the dependent files have to be uploaded in zip format that are required for the executable file to run.

5. Runtime Arguments

Run-time arguments can be provided if it is required in the program that is uploaded.

After filling up the required parameter, the ‘deploy’ button has to be clicked. The confirmation that the task is deployed will be visible as shown in the image below.

../_images/recurrentTaskDeploy3.png

Start Recurrent Task

After the task is deployed, the execution can be started by clicking the START button.

../_images/recurrentTaskStart1.png

Data files(.zip file) can be optionally uploaded and/or runtime arguments can be updated for the task execution.

../_images/recurrentTaskStart1.1.png

After clicking the start button the schedule status becomes RUNNING.

../_images/recurrentTaskStart2.png

At given regular intervals, the program execution will start and the status will be set to RUNNING.

../_images/recurrentTaskStart2.1.png

Once the execution of a particular recurrence is completed, the status is set to FINISHED. After the execution is complete, the output files generated,start time,end time and the time taken to complete the execution can be viewed.

../_images/recurrentTaskStart3.png

If a limited recurrence count is set, the schedule status is set to FINISHED after the count of execution is reached.

../_images/recurrentTaskStart4.png

Stop Recurrent Task

A particular recurrent task can be stopped by clicking the STOPTASK button.

../_images/recurrentTaskStop1.png

Once stopped the status of the recurrent task becomes ABORTED.

../_images/recurrentTaskStop2.png

Pause Recurrent Task

Particular recurrent task can be paused by clicking the PAUSETASK button.

../_images/recurrentTaskPause1.png

The status of the recurrent task becomes PAUSED.

../_images/recurrentTaskPause2.png

Resume Recurrent Task

Any particular recurrent task can be resumed by clicking the RESUMETASK button.

../_images/recurrentTaskResume1.png

The status of the recurrent task becomes RUNNING/QUEUED.

../_images/recurrentTaskResume2.png

Refresh a Recurrent Task

Any particular task can be refreshed by clicking the REFRESH button.

../_images/recurrentTaskRefresh1.png

Reschedule Recurrent Task

Any particular recurrent task can be rescheduled by clicking the RESCHEDULE button.

../_images/recurrentTaskReschedule1.png

On clicking the reschedule button the following form appears.

../_images/recurrentTaskReschedule2.1.png

1. Recurrent Task Name

This is the name of the task.

2. Cron Pattern

This allows to select the cron pattern of the recurrent task.

3. Counts

This allows to set the recurrence limit. If not set, i.e. set at 0, the task will schedule and run infinitely.

4. Select Upload of Input File

This gives the option of selecting whether or not to upload the input files.

5. Input Files(Optional)

This is the input files(.zip file) required for execution.

6. Runtime Arguments(Optional)

This is the runtime arguments for task execution.

../_images/recurrentTask3.png

The recurrent interval of the task can be as follows:

a. Second

b. Minute

c. Hour

d. Day

e. Week

f. Month

g. Year

Two types of cron pattern can be set:

1. Tasks can be run at a fixed time of a day,every day, or a fixed day and time of a week for every week, etc. For example, 7:30 AM every day, or 12:00 PM Monday, Tuesday, Wednesday every week, etc.

2. Task can be run at a regular interval of seconds or minutes. For example, every 15 minutes or every 30 seconds etc.

../_images/recurrentTaskReschedule3.png

Cancel Recurrent Task

Scheduling for a particular recurrent Task can be cancelled by clicking the CANCEL button.

../_images/recurrentTaskCancelScheduling1.png

The scheduling status is updated to CANCELLED.

../_images/recurrentTaskCancelScheduling2.png

Undeploy Recurrent Task

A particular recurrent task can be undeployed by clicking the UNDEPLOY button.

../_images/recurrentTaskUndeploy1.png

The status of the recurrent task becomes UNDEPLOYED.

../_images/recurrentTaskUndeploy1.png

Delete Recurrent Task

Any particular recurrent task can be deleted clicking the DELETE button.

../_images/recurrentTaskDelete1.png

A popup appears to confirm whether to proceed with the deletion.

../_images/recurrentTaskDelete2.png

Upon confirmation the recurrent task gets deleted successfully.

../_images/recurrentTaskDelete3.png

Note

  • Running a long running job in a recurrent task must be avoided.

  • Rescheduling of a recurrent task can only be done when the scheduling status is FINISHED/CANCELLED and the job state is ABORTED/FINISHED