Notebook Service

Introduction

The TCUP Notebook is an interactive web-based tool for data scientists to develop algorithms using Apache Spark on TCUP data source using. Using Pytcup library TCUP Notebook lets users write Python based code with Apache Spark, executed in a distributed cluster.

TCUP Notebook Service API’s allows tenants to spawn, manage multiple notebooks for multiple users. This service will onboard multiple data scientist into the platform to work with their scientific algorithms.

Features

Notebook Service has the following features:

  • Rest API’s to create/delete users and start/stop notebook servers .

  • GUI Utility for Tenant Admin to visualize the list of users at https://domainname.tcupiot.com/tcup/nb/notebook

  • Individual and isolated Docker based notebook environment for each user.

  • API to get detail of the user specific notebook.

  • Access the user specific notebook with valid user credential from browser at https://domainname.tcupiot.com/ns/hub/login

  • User develops Python based program using Pytcup library. For further detail please go through the Pytcup documentation.

  • Get the Spark Context from Spark Cluster.

  • Run algorithms in distributed Spark cluster using Spark Dataframes, RDD.

  • Enabled dynamic resource allocation within Spark cluster.

  • Python based Machine Learning programs using distributed SparkML, TensorFlow etc can be leveraged.

  • Visualize the analytics output using Pandas, matplotlib etc.

User Hierarchy and Credentials

TCUP Notebook Service API can be called with x-api-key (TCUP Tenant x-api-key)

Three levels of hierarchy are provided:

  • TCUP Admin (Super Admin) can Create or Delete Tenants and Notebook Users under that Tenant. Access to TCUP Admin credentials is only with TCUP Portal Service.

  • Tenant Admin (Same as Tenant User in TCUP Portal) can create the Notebook users using TCUP IAM Service. TCUP credential to be used to login to user notebook. Tenant Admin can start/stop Notebook user Dockers (Using Notebook service’s User-Server API ). A GUI utility for Tenant Admin to create/delete Users and Start/stop Notebook Dockers is provided at https://domainname.tcupiot.com/tcup/nb/notebook

  • Notebook User (Data Scientist/Developer) uses Jupyter Notebook, pytcup, Spark for data analysis/building Applications. After a user is created user can log in to notebook service at https://domainname.tcupiot.com/ns/hub/login Then use Jupyter Notebooks and pytcup library and connect to Spark cluster from Jupyter Notebook. He can start/stop his server and Logout using GUI of Jupyter Notebook. This Notebook user has the least privileges i.e he can only obtain his own Notebook Docker Server Status (whether the server has started or stopped ) and can stop/start his own notebook server using Rest API also.

Usage Overview

  • Step1: Tenant Admin logs in to Notebook UI, using TCUP Portal credentials at https://domainname.tcupiot.com/

../../_images/Tenant_Admin.png

  • Step2: Tenant Admin creates Notebook User and receives user login credentials using the TCUP IAM service categorized under Services, at https://domainname.tcupiot.com/tcup/nb/notebook

../../_images/Tenant_admin_GUI.png

  • Step3: Notebook User visits the Notebook Login Page at https://domainname.tcupiot.com/ns/hub/login and logs in via User credentials

../../_images/Notebook_login.png

  • Step4: After login, initial home screen can be viewed. Like a file browser, this is where all notebooks and files can be browsed.

../../_images/Home_screen.png

  • Step5: New python3 Notebook can be created

../../_images/Create_new_notebook.png

  • Step6: Blank notebook with name Untitled will start

../../_images/New_notebook.png

  • Step7: Plotting and python3 code execution is possible in notebook cells. Import pytcup python package to interface with TCUP Services.

../../_images/Plotting.png

  • Step8: Save your changes and Download/Share your Notebook as (.ipynb/.py/.html/.pdf/.tex/.rst/.md) format

../../_images/FileMenu.png

  • Step9: All running notebooks can be viewed and shutdown in Running Tab on Home screen

../../_images/Running_notebooks.png

  • Step10: Stop/Start Notebook Docker Server via Control Panel before logout

../../_images/ControlPanel.png