Namespaces

Variants
Actions
Personal tools

Jupyter

From UABgrid Documentation

(Difference between revisions)
Jump to: navigation, search
(putty port forward link added)
(Jupyter On Demand)
 
(24 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 
[http://jupyter.org/ Jupyter Notebook]  is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text. For more information on jupyter notebook, click [http://jupyter.org/documentation here].
 
[http://jupyter.org/ Jupyter Notebook]  is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text. For more information on jupyter notebook, click [http://jupyter.org/documentation here].
  
== Jupyter on Cheaha ==
+
= Jupyter On Demand =
 +
 
 +
As of 2019, UAB Research Computing allows access to cheaha via [http://rc.uab.edu On Demand]. To access.
 +
 
 +
== 1. Click [http://rc.uab.edu On Demand] ==
 +
 
 +
 
 +
== 2. Select Interactive App and pick Jupyter Notebook ==
 +
 
 +
[[File:JupyterNotebookStart.png|500px]]
 +
 
 +
== 3. Load in Anaconda ==
 +
'''
 +
module load Anaconda3/5.3.1
 +
'''
 +
The following should also work for an updated version of Anaconda.
 +
'''
 +
module load Anaconda3
 +
'''
 +
 
 +
== 4. If you require running on a '''''GPU''''', please add the following to your environment. ==
 +
 
 +
'''
 +
module load cuda92/toolkit/9.2.88
 +
module load CUDA/9.2.88-GCC-7.3.0-2.30
 +
'''
 +
 
 +
Additionally, you will need to request a GPU as shown below by
 +
including the pascalenodes argument:
 +
 
 +
[[File:PascalNodes.png|500px]]
 +
 
 +
== 5. Click Launch ==
 +
 
 +
[[File:LaunchJupyter.png|500px]]
 +
 
 +
Wait until you receive an email or get a blue Launch button. This can happen in about 10-20 seconds or may take much longer depending on the resources (CPU count and memory requested).
 +
 
 +
== 6. Connect to Jupyter Notebook ==
 +
 
 +
== 7. Test Pytorch with a new notebook ==
 +
 
 +
[[File:TestPytorch.png|500px]]
 +
 
 +
= Jupyter by Proxy =
 +
'''''(not longer required as of August 2019, use OnDemand option instead and this only as a fallback option)'''''
  
 
The cheaha cluster supports Jupyter notebooks for data analysis, but such jobs should be running using the SLURM job submission system to avoid overloading the head node. To run a Jupyter Notebook on cheaha, login to cheaha from your client machine and start an [https://docs.uabgrid.uab.edu/wiki/Slurm#Interactive_Job interactive job].  
 
The cheaha cluster supports Jupyter notebooks for data analysis, but such jobs should be running using the SLURM job submission system to avoid overloading the head node. To run a Jupyter Notebook on cheaha, login to cheaha from your client machine and start an [https://docs.uabgrid.uab.edu/wiki/Slurm#Interactive_Job interactive job].  
Line 7: Line 52:
 
One important note is that cheaha only supports openssh, you should be able to use native ssh from Mac or Linux machines. Windows 10 supports openssh as well, but it is not enabled by default. On updated Windows 10 machines, '''a Developers Command Prompt''' (available via searching from the Start Menu) is able to run openssh via the ssh command similar to Mac and Linux users. Another option for Windows machines is the installation of Cygwin. Putty has been  [[Setting_Up_VNC_Session#Port-forwarding_from_Windows_Systems|tested]], but does not work reliably on cheaha for proxying connections.
 
One important note is that cheaha only supports openssh, you should be able to use native ssh from Mac or Linux machines. Windows 10 supports openssh as well, but it is not enabled by default. On updated Windows 10 machines, '''a Developers Command Prompt''' (available via searching from the Start Menu) is able to run openssh via the ssh command similar to Mac and Linux users. Another option for Windows machines is the installation of Cygwin. Putty has been  [[Setting_Up_VNC_Session#Port-forwarding_from_Windows_Systems|tested]], but does not work reliably on cheaha for proxying connections.
  
 +
The Jupyter notebooks is built with [[Anaconda]],a free and open source distribution of python and R for scientific computing. If you need additional packages, you can create your own [[Python_Virtual_Environment]] just for that purpose.
 +
 +
== 1. Start the Jupyter Notebook ==
 
<pre>
 
<pre>
 
srun --ntasks=1 --cpus-per-task=4 --mem-per-cpu=4096 --time=08:00:00 --partition=medium --job-name=JOB_NAME --pty /bin/bash
 
srun --ntasks=1 --cpus-per-task=4 --mem-per-cpu=4096 --time=08:00:00 --partition=medium --job-name=JOB_NAME --pty /bin/bash
Line 13: Line 61:
 
jupyter notebook --no-browser --ip=$host
 
jupyter notebook --no-browser --ip=$host
 
</pre>
 
</pre>
Additionally, if anticipating large IO data transfer adjust the run command to set a higher data rate limit as shown below:
+
A headless Jupyter notebook should now be running on a compute node. The next step is to proxy this connection to your local machine.
 +
 
 +
== 2. Proxy Connection Locally ==
 +
Now, start up a '''new''' tab/terminal/window on your client machine and relogin to cheaha, using
 
<pre>
 
<pre>
jupyter notebook --no-browser --ip=$host --NotebookApp.iopub_data_rate_limit=1.0e10
+
ssh -L 88XX:c00XX:88XX BLAZERID@cheaha.rc.uab.edu
 
</pre>
 
</pre>
 +
'''Note:'''
 +
* '''c00XX''' is the compute node where you started the jupyter notebook, for example c0047
 +
* '''88XX''' is the port that the notebook is running, for example 8888
 +
* For windows users, you can find instructions for port forwarding, [https://docs.uabgrid.uab.edu/wiki/Setting_Up_VNC_Session#Port-forwarding_from_Windows_Systems here]
  
After running the jupyter notebook command the server should start running and provide you with a URL including a port # (typically but not always 8888) and a compute node on cheaha (for example C0047) that looks something like this:
+
== 3. Copy notebook URL ==
 +
After running the jupyter notebook command the server should start running in headless mode and provide you with a URL including a port # (typically but not always 8888) and a compute node on cheaha (for example C0047) that looks something like this:
 
<pre>
 
<pre>
 
 
 
     Copy/paste this URL into your browser when you connect for the first time,
 
     Copy/paste this URL into your browser when you connect for the first time,
 
     to login with a token:
 
     to login with a token:
 
         http://c0047:8888/?token=73da89e0eabdeb9d6dc1241a55754634d4e169357f60626c&token=73da89e0eabdeb7d6dc1241a55754634d4e169357f60626c
 
         http://c0047:8888/?token=73da89e0eabdeb9d6dc1241a55754634d4e169357f60626c&token=73da89e0eabdeb7d6dc1241a55754634d4e169357f60626c
 
</pre>
 
</pre>
 +
Copy the URL shown below into you clipboard/buffer for pasting into the browser as shown in step 4).
  
Now, start up a new tab/terminal/window on your client machine, and relogin to cheaha, using
+
== 4. Access Notebook through Local Browser via Proxy Connection ==
 +
Now access the link on your client machine browser locally using the link generated by jupyter notebook by '''substituting in localhost instead of c00XX'''. Make sure you have the correct port as well.
 
<pre>
 
<pre>
ssh -L 88XX:c00XX:88XX BLAZERID@cheaha.rc.uab.edu
+
http://localhost:88XX/?token=73da89e0eabdeb9d6dc1241a55754634d4e169357f60626c&token=73da89e0eabdeb7d6dc1241a55754634d4e169357f60626c
 
</pre>
 
</pre>
'''Note:'''
+
A Jupyter notebook should then open in your browser connected to the compute node.
* '''c00XX''' is the compute node where you started the jupyter notebook, for example c0047
+
* '''88XX''' is the port that the notebook is running, for example 8888
+
  
Now access the link on your client machine browser locally using the link generated by jupyter notebook by '''substituting in localhost instead of c00XX'''. Make sure you have the correct port as well.
+
== Jupyter Options ==
 +
 
 +
=== DeepNLP option (development in progress) ===
 +
For the use of additional libraries (pytorch, spacy) related to Deep Learning and/or NLP after loading Anaconda3/5.2.0 run:
 
<pre>
 
<pre>
http://localhost:88XX/?token=73da89e0eabdeb9d6dc1241a55754634d4e169357f60626c&token=73da89e0eabdeb7d6dc1241a55754634d4e169357f60626c
+
conda activate /share/apps/rc/software/Anaconda3/5.2.0/envs/DeepNLP
 
</pre>
 
</pre>
  
A Jupyter notebook should then open in your browser.
+
=== Heavy Data IO option ===
 +
Additionally, if anticipating large IO data transfer adjust the run command to set a higher data rate limit as shown below:
 +
<pre>
 +
jupyter notebook --no-browser --ip=$host --NotebookApp.iopub_data_rate_limit=1.0e10
 +
</pre>
 +
 
 +
=== Memory Heavy option ===
 +
<pre>
 +
srun --ntasks=1 --cpus-per-task=4 --mem-per-cpu=16384 --time=08:00:00 --partition=medium --job-name=POSTag --pty /bin/bash
 +
</pre>
 +
 
 +
=== GPU Option ===
 +
Finally, if your job requires a GPU then add the [https://docs.uabgrid.uab.edu/wiki/Slurm#Requesting_for_GPUs gres and partition arguments] as shown below:
 +
<pre>
 +
srun --ntasks=1 --cpus-per-task=1 --mem-per-cpu=4096 --time=08:00:00 --partition=pascalnodes --job-name=JOB_NAME --gres=gpu:1 --pty /bin/bash
 +
</pre>

Latest revision as of 20:08, 27 October 2019

Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text. For more information on jupyter notebook, click here.

Contents

[edit] Jupyter On Demand

As of 2019, UAB Research Computing allows access to cheaha via On Demand. To access.

[edit] 1. Click On Demand

[edit] 2. Select Interactive App and pick Jupyter Notebook

Error creating thumbnail: convert: unable to open image `/research/apps/rnb/docs/htdocs/tgw/images/5/56/JupyterNotebookStart.png': No such file or directory @ error/blob.c/OpenBlob/2589.
convert: unable to open file `/research/apps/rnb/docs/htdocs/tgw/images/5/56/JupyterNotebookStart.png' @ error/png.c/ReadPNGImage/3639.
convert: missing an image filename `/tmp/transform_1e888ab-1.png' @ error/convert.c/ConvertImageCommand/3015.

[edit] 3. Load in Anaconda

module load Anaconda3/5.3.1

The following should also work for an updated version of Anaconda.

module load Anaconda3

[edit] 4. If you require running on a GPU, please add the following to your environment.

module load cuda92/toolkit/9.2.88
module load CUDA/9.2.88-GCC-7.3.0-2.30

Additionally, you will need to request a GPU as shown below by including the pascalenodes argument:

Error creating thumbnail: convert: unable to open image `/research/apps/rnb/docs/htdocs/tgw/images/5/57/PascalNodes.png': No such file or directory @ error/blob.c/OpenBlob/2589.
convert: unable to open file `/research/apps/rnb/docs/htdocs/tgw/images/5/57/PascalNodes.png' @ error/png.c/ReadPNGImage/3639.
convert: missing an image filename `/tmp/transform_5a8174e-1.png' @ error/convert.c/ConvertImageCommand/3015.

[edit] 5. Click Launch

Error creating thumbnail: convert: unable to open image `/research/apps/rnb/docs/htdocs/tgw/images/8/81/LaunchJupyter.png': No such file or directory @ error/blob.c/OpenBlob/2589.
convert: unable to open file `/research/apps/rnb/docs/htdocs/tgw/images/8/81/LaunchJupyter.png' @ error/png.c/ReadPNGImage/3639.
convert: missing an image filename `/tmp/transform_25d7af8-1.png' @ error/convert.c/ConvertImageCommand/3015.

Wait until you receive an email or get a blue Launch button. This can happen in about 10-20 seconds or may take much longer depending on the resources (CPU count and memory requested).

[edit] 6. Connect to Jupyter Notebook

[edit] 7. Test Pytorch with a new notebook

Error creating thumbnail: convert: unable to open image `/research/apps/rnb/docs/htdocs/tgw/images/5/56/TestPytorch.png': No such file or directory @ error/blob.c/OpenBlob/2589.
convert: unable to open file `/research/apps/rnb/docs/htdocs/tgw/images/5/56/TestPytorch.png' @ error/png.c/ReadPNGImage/3639.
convert: missing an image filename `/tmp/transform_44b9647-1.png' @ error/convert.c/ConvertImageCommand/3015.

[edit] Jupyter by Proxy

(not longer required as of August 2019, use OnDemand option instead and this only as a fallback option)

The cheaha cluster supports Jupyter notebooks for data analysis, but such jobs should be running using the SLURM job submission system to avoid overloading the head node. To run a Jupyter Notebook on cheaha, login to cheaha from your client machine and start an interactive job.

One important note is that cheaha only supports openssh, you should be able to use native ssh from Mac or Linux machines. Windows 10 supports openssh as well, but it is not enabled by default. On updated Windows 10 machines, a Developers Command Prompt (available via searching from the Start Menu) is able to run openssh via the ssh command similar to Mac and Linux users. Another option for Windows machines is the installation of Cygwin. Putty has been tested, but does not work reliably on cheaha for proxying connections.

The Jupyter notebooks is built with Anaconda,a free and open source distribution of python and R for scientific computing. If you need additional packages, you can create your own Python_Virtual_Environment just for that purpose.

[edit] 1. Start the Jupyter Notebook

srun --ntasks=1 --cpus-per-task=4 --mem-per-cpu=4096 --time=08:00:00 --partition=medium --job-name=JOB_NAME --pty /bin/bash
module load Anaconda3/5.2.0
unset XDG_RUNTIME_DIR
jupyter notebook --no-browser --ip=$host

A headless Jupyter notebook should now be running on a compute node. The next step is to proxy this connection to your local machine.

[edit] 2. Proxy Connection Locally

Now, start up a new tab/terminal/window on your client machine and relogin to cheaha, using

ssh -L 88XX:c00XX:88XX BLAZERID@cheaha.rc.uab.edu

Note:

  • c00XX is the compute node where you started the jupyter notebook, for example c0047
  • 88XX is the port that the notebook is running, for example 8888
  • For windows users, you can find instructions for port forwarding, here

[edit] 3. Copy notebook URL

After running the jupyter notebook command the server should start running in headless mode and provide you with a URL including a port # (typically but not always 8888) and a compute node on cheaha (for example C0047) that looks something like this:

    Copy/paste this URL into your browser when you connect for the first time,
    to login with a token:
        http://c0047:8888/?token=73da89e0eabdeb9d6dc1241a55754634d4e169357f60626c&token=73da89e0eabdeb7d6dc1241a55754634d4e169357f60626c

Copy the URL shown below into you clipboard/buffer for pasting into the browser as shown in step 4).

[edit] 4. Access Notebook through Local Browser via Proxy Connection

Now access the link on your client machine browser locally using the link generated by jupyter notebook by substituting in localhost instead of c00XX. Make sure you have the correct port as well.

http://localhost:88XX/?token=73da89e0eabdeb9d6dc1241a55754634d4e169357f60626c&token=73da89e0eabdeb7d6dc1241a55754634d4e169357f60626c

A Jupyter notebook should then open in your browser connected to the compute node.

[edit] Jupyter Options

[edit] DeepNLP option (development in progress)

For the use of additional libraries (pytorch, spacy) related to Deep Learning and/or NLP after loading Anaconda3/5.2.0 run:

conda activate /share/apps/rc/software/Anaconda3/5.2.0/envs/DeepNLP

[edit] Heavy Data IO option

Additionally, if anticipating large IO data transfer adjust the run command to set a higher data rate limit as shown below:

jupyter notebook --no-browser --ip=$host --NotebookApp.iopub_data_rate_limit=1.0e10 

[edit] Memory Heavy option

srun --ntasks=1 --cpus-per-task=4 --mem-per-cpu=16384 --time=08:00:00 --partition=medium --job-name=POSTag --pty /bin/bash

[edit] GPU Option

Finally, if your job requires a GPU then add the gres and partition arguments as shown below:

srun --ntasks=1 --cpus-per-task=1 --mem-per-cpu=4096 --time=08:00:00 --partition=pascalnodes --job-name=JOB_NAME --gres=gpu:1 --pty /bin/bash