From UABgrid Documentation
Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text. For more information on jupyter notebook, click here.
Jupyter on Cheaha
The cheaha cluster supports Jupyter notebooks for data analysis, but such jobs should be running using the SLURM job submission system to avoid overloading the head node. To run a Jupyter Notebook on cheaha, login to cheaha from your client machine and start an interactive job.
One important note is that cheaha only supports openssh, you should be able to use native ssh from Mac or Linux machines. Windows 10 supports openssh as well, but it is not enabled by default. On updated Windows 10 machines, a Developers Command Prompt (available via searching from the Start Menu) is able to run openssh via the ssh command similar to Mac and Linux users. Another option for Windows machines is the installation of Cygwin. Putty has been tested, but does not work reliably on cheaha for proxying connections.
Starting the Jupyter Notebook
srun --ntasks=1 --cpus-per-task=4 --mem-per-cpu=4096 --time=08:00:00 --partition=medium --job-name=JOB_NAME --pty /bin/bash module load Anaconda3/5.2.0 unset XDG_RUNTIME_DIR jupyter notebook --no-browser --ip=$host
DeepNLP option (development in progress)
For the use of additional libraries (pytorch, spacy) related to Deep Learning and/or NLP after loading Anaconda3/5.2.0 run:
conda activate /share/apps/rc/software/Anaconda3/5.2.0/envs/DeepNLP
Heavy Data IO option
Additionally, if anticipating large IO data transfer adjust the run command to set a higher data rate limit as shown below:
jupyter notebook --no-browser --ip=$host --NotebookApp.iopub_data_rate_limit=1.0e10
Memory Heavy option
srun --ntasks=1 --cpus-per-task=4 --mem-per-cpu=16384 --time=08:00:00 --partition=medium --job-name=POSTag --pty /bin/bash
Finally, if your job requires a GPU then add the gres and partition arguments as shown below:
srun --ntasks=1 --cpus-per-task=1 --mem-per-cpu=4096 --time=08:00:00 --partition=pascalnodes --job-name=JOB_NAME --gres=gpu:1 --pty /bin/bash
Copy notebook settings
After running the jupyter notebook command the server should start running and provide you with a URL including a port # (typically but not always 8888) and a compute node on cheaha (for example C0047) that looks something like this:
Copy/paste this URL into your browser when you connect for the first time, to login with a token: http://c0047:8888/?token=73da89e0eabdeb9d6dc1241a55754634d4e169357f60626c&token=73da89e0eabdeb7d6dc1241a55754634d4e169357f60626c
Proxy Connection Locally
Now, start up a new tab/terminal/window on your client machine and relogin to cheaha, using
ssh -L 88XX:c00XX:88XX BLAZERID@cheaha.rc.uab.edu
- c00XX is the compute node where you started the jupyter notebook, for example c0047
- 88XX is the port that the notebook is running, for example 8888
Access Notebook through Local Browser via Proxy Connection
Now access the link on your client machine browser locally using the link generated by jupyter notebook by substituting in localhost instead of c00XX. Make sure you have the correct port as well.
A Jupyter notebook should then open in your browser.