MATLAB DCS: Difference between revisions

From Cheaha
Jump to navigation Jump to search
No edit summary
No edit summary
 
(132 intermediate revisions by 7 users not shown)
Line 1: Line 1:
__TOC__
{{Matlab_deprecated}}
Cheaha now has a 128 node license for the [http://www.mathworks.com/products/distriben/ Distributed Computing Server] component of Matlab.


In order to use DCS on Cheaha, you will have to use your own Matlab and [http://www.mathworks.com/products/parallel-computing/ Parallel Computing Toolbox] license.


== SSH Keys ==
The [http://www.mathworks.com/products/distriben MATLAB Distributed Computing Server (MATLAB DCS)] is a parallel computing extension to MATLAB that enables processing to be spread across a large number of worker nodes, accelerating the speed at which compute intensive operations can complete.


The Matlab Parallel Toolbox uses SSH authentication via public-private key-pair to connect to the Matlab Distributed Computing Server on the cluster head node.
UAB IT Research Computing maintains a 128 worker node license for the the [[Cheaha]] computing platform. In order to use DCS on Cheaha, you will need to [[MATLAB#Using_MATLAB|use a MATLAB instance]] with the Parallel Computing Toolbox installed.


The process of configuring SSH keys differs depends on your client operating system. Linux and Mac have the appropriate SSH client software, Windows will require the installation of PuTTY.
In order to leverage the MATLAB worker nodes on [[Cheaha]] the Parallel Computing Toolbox will need to be configured to submit compute tasks to [[Cheaha]] by following the steps in this document.
=== Windows ===


This section documents the steps to install and configure PuTTY on Windows computers. This software provides the utilities that Matlab uses to communicate with the head node.
{{MatlabAppPage}}


==== PuTTY ====
== Overview ==
If PuTTY isn't already installed on your system, download the tools from the [http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html PuTTY Downloads page].


Download and run [http://the.earth.li/~sgtatham/putty/latest/x86/putty-0.60-installer.exe this file] to install PuTTY using the graphical installation tool.
The following outline highlights the steps involved to configure your MatLab install and write programs that submit tasks to the worker nodes of the Distributed Computing Server on Cheaha:


Alternatively, you can download the individual components to a directory of your choosing:
* Configure the Task Submit Environment (One-time Setup)
* [http://the.earth.li/~sgtatham/putty/latest/x86/putty.exe putty.exe]
** [[MatLab|Install MatLab]] with the Parallel Computing Toolbox on your Windows / Linux / Mac workstation
* [http://the.earth.li/~sgtatham/putty/latest/x86/plink.exe plink.exe]
** Download and extract the MatLab task submission functions to your workstation MatLab environment
* [http://the.earth.li/~sgtatham/putty/latest/x86/pageant.exe pageant.exe]
** Define the "cheaha" parallel configuration in your workstation MatLab environment to submit tasks to Cheaha
* [http://the.earth.li/~sgtatham/putty/latest/x86/puttygen.exe puttygen.exe]
** Run the validation tests to ensure your "cheaha" parallel configuration works
* [http://the.earth.li/~sgtatham/putty/latest/x86/pscp.exe pscp.exe]
* Develop and Run Parallel Computing Applications
* [http://the.earth.li/~sgtatham/putty/latest/x86/psftp.exe psftp.exe]
** Write, test and debug your parallel code on your local workstation using the default "local" parallel configuration
** Once your code works, select the "cheaha" parallel configuration to submit tasks to the Cheaha cluster. Note: your workstation MatLab application does not need to keep running after the tasks are submitted.
** You will receive an email when the tasks you submitted are complete
** Use your workstation MatLab application to retrieve the results
** When you are finished with your job contexts, clean up the job related content to free disk space


===== Generate an SSH Key Pair =====
== Using MATLAB DCS ==


Generate a public-private key pair by running the puttygen command.
The MATLAB Distributed Computing Services (DCS) are accessed via the Parallel Computing Toolbox (PCT) which is installed as part of your desktop [[MATLAB|MATLAB installation]]. The PCT allows MATLAB running on your workstation to send MATLAB code and data (tasks) to the cluster directly from the comfort of your familiar MATLAB environment on your desktop. This makes the expanded compute power of [[Cheaha]] available to processes work loads that exceed the capabilities of your desktop computer.  Once your tasks are submitted to [[Cheaha]], your desktop MATLAB is also free to move on to other tasks or be closed completely, freeing your desktop or laptop for your other activities.


Start PuTTYgen by either:
Configuring the Parallel Computing Toolbox involves three steps documented below:
* Clicking Start -> All Programs -> PuTTY --> PuTTYgen
# install MATLAB submit functions on your workstation
* Opening Windows Explorer / My Computer, browse to the PuTTY directory and double click 'puttygen'
# configure the "cheaha" parallel computing target to which PCT tasks can be submitted
# run the validation tests to confirm a working installation.


This will bring up a window to manage your key pair.
This page documents the DCS configuration for MATLAB 2010b and later. For DCS configuration instructions on previous versions of MATLAB, please see the page [[MatLab DCS R2010a and Earlier]]


[[Image:puttygen-window.png|400px]]
Using MATLAB DCS requires you have a cluster account on [[Cheaha]].  Please request an account by sending an email to [[mailto:support@listserv.uab.edu]] and include your campus affiliation and a brief statement of your research interests for using the cluster. 


Press the "Generate" button to start the process.  You will be requested to move your mouse around in the blank are of that window to help generate a good random number.  The progress bar will fill in as you do so letting you know when the process is complete.  Once complete, the PuTTY Gen window will display your public key and offer you various options to work with that key. 
=== MATLAB DCS from Your Desktop ===


[[Image:puttygen-savekey2.png|400px]]
==== MATLAB Submit Functions ====


# Change the "Key Comment" to your Windows host name (run 'hostname' at the command prompt to discover the name of your Windows system)
The MATLAB submit functions create a cluster job context for your code and are responsible for transferring your code and the data it analyzes to the cluster for processing.
# Set a passphrase for your private key, filling in both the "Key passphrase" and "Confirm passphrase" text boxes with the same passphrase.  The passphrase is a '''local''' password for this private key.  It doesn't have anything to do with any other passwords.  It is strictly about protecting the private key that you just generated. Please refer to the UAB IT page for instructions on [http://main.uab.edu/Sites/it/faqs/49118/ creating a strong password]. You need to remember this passphrase because you will be prompted for it whenever you use this key-pair.
# Press the "Save private key" button. The save button, will by default save the private key to your "My Documents" folder.  This is fine, and you can give any file name you like.  Just remember the name and where you saved it, so you can load it in the next steps. 
# Keep your PuTTY Key Generator window open so we can use below when we register you public key with the SSH server.
#'''Remember your passphrase.'''


===== Create a Session Definition =====
These submit functions must be installed on your computer and must be accessible to MATLAB via the MATLAB PATH environment.  The easiest way to accomplish this is to copy the submit functions to the default directory created for by MATLAB.  These directories on the respective operating systems are listed below.


The PuTTY tools that Matlab Parallel Toolbox leverages are configured by creating "Saved Sessions" PuTTY.
All operating systems (Windows, Linux and Mac) are supported by the same set of submit functions. The functions are written in MATLAB making them  cross-platform and only dependent on the version of MATLAB in use.


To create a PuTTY session for cheaha.uabgrid.uab.edu, follow these steps.
# Download the MATLAB submit functions
#* [http://projects.uabgrid.uab.edu/matlab/browser/trunk/distributables/matlab-R2013a-nonshared.zip?format=raw Submit Functions for MATLAB R2013a] -(updated 09/04/2013)
#* [http://projects.uabgrid.uab.edu/matlab/browser/trunk/distributables/matlab-R2012a-nonshared.zip?format=raw Submit Functions for MATLAB R2012a] -(updated 03/07/2012)
#* [http://projects.uabgrid.uab.edu/matlab/browser/trunk/distributables/matlab-R2011b-nonshared.zip?format=raw Submit Functions for MATLAB R2010b, R2011a, R2011b] -(updated 02/21/2011)
# Unzip the files to a directory included in your MATLAB PATH setting. Recommended locations are:
#* Windows:  <pre>My Documents\MATLAB</pre>
#* Linux:    <pre>$HOME/Documents/MATLAB</pre>
#* Mac:      <pre>$HOME/Documents/MATLAB</pre>


Start PuTTY by either:
Once the submit function files have been downloaded and unzipped in the above paths, restart MATLAB to ensure they are properly loaded in your environment.
* Clicking Start -> All Programs -> PuTTY --> PuTTY
* Opening Windows Explorer / My Computer, browse to the PuTTY directory and double click 'putty'


# This brings up the following dialog that has a collection of configuration categories in the "Category:" window on the left, and a context sensitive set of actions on the right.  That is, if you change the category on the left you will change what you see on the right. The default category that you see is the Session, which is the main focus of our work here. <br>[[Image:putty-session-dialog.png|400px]]
NOTE: If you choose not to use the above path recommendations, your MATLAB PATH may be viewed/altered by starting the MATLAB client on your workstation and clicking File -> Set Path and adding the path in which you unpacked the submit functions.
# In the default "Session" category, fill in the "Host Name (or IP address)" text box with "cheaha.uabgrid.uab.edu", and fill in the "Saved Sessions" text box with "cheaha.uabgrid.uab.edu".  Press the "Save" button <br>[[Image:putty-session-dialog-filled-cheaha.png|400px]]
# Select the "Data" sub-category under the "Connection" category from the left-side Category browser.  Fill in the "Auto-login username" text box with the username for your account on cheaha.
# Select the "Session" category, and press "Save" again.  You now have a saved session ready for use.
# Press the "Open" button at the bottom on the dialog. This will bring up a login window with a login prompt for your password.  Provide your cheaha password to login.  Keep this session open, as you will use it in the next section.


===== Register Your Public Key =====
==== Parallel Computing Toolbox Configuration ====


In order to use your public-private key-pair to start an SSH session, you need to register you public key with cheaha by adding your public key to the list of authorized keys for that SSH server.  
The Parallel Computing Toolbox (PCT) enables language extensions in MATLAB that support dividing your application into tasks that can be executed in parallel.  By default, all of these tasks will run on your local workstation using the pre-defined "local" PCT configuration.


# Use the SSH connection established in the previous step
To run these tasks on the Cheaha compute cluster, a new configuration for the PCT must be defined. In this section we will create the "cheaha" configuration and run a quick validation test to confirm its operation.
# In the PuTTYgen window text area labeled "Public key for pasting into OpeSSH authorized_keys file:", select the public key by right clicking the key and clicking "Select All". Copy the key by right clicking again and selecting "Copy"
# In the SSH session window, enter the command <pre>vi $HOME/.ssh/authorized_keys</pre>
# Press the '''SHIFT''' and '''o''' keys (in other words, a capital letter O), and the press your '''right''' mouse button over this window. This will paste your public key onto a new line.  Press the '''Escape''' (Esc) key, press the key sequence colon-w-q-enter (:wq Enter)
# Fix the file permissions using the command <pre>chmod u=rw $HOME/.ssh/authorized_keys</pre>
# End the session. Type the command <pre>exit</pre>


You can close the PuTTY Configuration and PuTTY Key Generator windows now if you haven't already done so.
'''Prior to continuing''', make sure you:
* can establish an SSH connection to Cheaha
* have followed the steps in the previous section


===== Load Your Private Key =====
===== Create the "cheaha" PCT Configuration =====
Download and save the Cheaha cluster configuration file for your MATLAB version
# [http://projects.uabgrid.uab.edu/matlab/browser/trunk/parallel-configs/cheaha-R2011b.mat?format=raw R2010b, R2011a, R2011b] cluster configuration file
#* Start MATLAB on your workstation
#* Click the "Parallel" menu
#* Click "Manage Configurations"
#* In the "Configurations Manager" window, click "File -> Import"
#* Browse to the location where you saved the '''cheaha-R2011b.mat''' file, select it, and click "Open"
# [http://projects.uabgrid.uab.edu/matlab/browser/trunk/parallel-configs/cheaha-R2012a.settings?format=raw R2012a] cluster configuration file
#* Start MATLAB R2012a on your workstation
#* Click the "Parallel" menu
#* Click "Manage Cluster Profiles"
#* In the "Cluster Profile Manager" window, click the "Import" button on the toolbar
#* Browse to the location where you saved the '''cheaha-R2012a.settings''' file, select it, and click "Open"


Loading your private key is the first step you will take prior to using Matlab to submit jobs to cheaha.  This step essentially activates your key so that all PuTTY-based tools can use it in their operations.
The Configuration Manager for R2011b and prior should now list a new entry named "cheaha" as shown in the following image, R2012a and later will also have a new entry in the Cluster Profile list:
[[Image:2011_config_mngr.png|none|x400px]]


Start PuTTYagent by either:
===== Personalize the "cheaha" PCT Configuration -2011b and earlier =====
* Clicking Start -> All Programs -> PuTTY --> PuTTYagent
* Opening Windows Explorer / My Computer, browse to the PuTTY directory and double click 'pagent'


This will place the "Pageant" icon (a computer wearing a tilted hat: [[Image:pageant-icon.png]]) in you shortcuts toolbar in the lower-right of our screen.
#Double click on cheaha in the Configuration Manager window to open the configuration editor. (Note: stretch the "Generic Scheduler Configuration Properties" window to the right so that you can view all of the text in the fields making it easier to read and edit correctly.)
# Click on the icon to bring up the Pageant Key List dialog <br> [[Image:pageant-dialog.png|300px]]
# Edit the following fields to use your personal data directories
# Press the "Add Key" button.  From the file dialog, select the private key created above (in My Documents by default)
#* '''ClusterMatlabRoot''': Make sure that the Root directory of MATLAB installation for workers matches the exact version of MATLAB you are using on your workstation. In this example '''/share/apps/mathworks/R2011a''' matches a MATLAB R2011a workstation install.  Change the "R2011a" to match your workstation MATLAB version.
# Press "Open" on the file dialog, you will be prompted for the passphrase to load your private key. Enter in the passphrase and click OK
#* '''DataLocation'''    : Change the directory path where job data is stored to an existing directory on your workstation where MATLAB can stage job files.
#* '''ParallelSubmitFcn''': Change the text "YOURUSERID" to your login id on Cheaha
#* '''SubmitFcn'''        : Change the text "YOURUSERID" to your login id on Cheaha
# Click 'OK'to save the configuration
# SSH to cheaha and make sure to create the $USER_SCRATCH/matlab directory. If this directory does not exist, the parallel computing toolbox jobs will fail.


The private key should now be loaded and ready for use with all PuTTY-based tools. Click the close button to minimize PuTTYagent.
The initial configuration will look similar to this screen shot.  You will need to edit the fields as describe in the preceding steps before you can use the configuration.  '''NOTE: be sure to replace the template user name settings "YOURUSERNAME" with the appropriate settings for your desktop and cluster account.'''


Warning: While Pagent is running and has your private key loaded, anyone else using your computer can access your account on Cheaha. Please LOCK your workstation when unattended!
[[Image:Cheaha_parallel_config.png|none|x650px]]


You are now ready to begin using the Matlab Parallel Toolbox.
===== Personalize the "cheaha" PCT Configuration -2012a  =====


=== Linux ===
#Double click on cheaha in the Configuration Manager window to open the configuration editor. (Note: stretch the "Generic Scheduler Configuration Properties" window to the right so that you can view all of the text in the fields making it easier to read and edit correctly.)
# If you don't already have an RSA key generated for your workstation, do so by running the following command (choose a good passphrase)
# Edit the following fields to use your personal data directories
<pre>$ ssh-keygen -t rsa
#* '''ClusterMatlabRoot''': Make sure that the Root directory of MATLAB installation for workers matches the exact version of MATLAB you are using on your workstation. In this example '''/share/apps/mathworks/R2012a''' matches a MATLAB R2012a workstation install.  Change the "R2012a" to match your workstation MATLAB version.
#* '''DataLocation'''    : Change the directory path where job data is stored to an existing directory on your workstation where MATLAB can stage job files.
#* '''independentSubmitFcn''': Change the text "YOURUSERID" to your login id on Cheaha
#* '''communicatingSubmitFcn'''        : Change the text "YOURUSERID" to your login id on Cheaha
# Click 'OK'to save the configuration
# SSH to cheaha and make sure to create the $USER_SCRATCH/matlab directory. If this directory does not exist, the parallel computing toolbox jobs will fail.


Enter file in which to save the key (~/.ssh/id_rsa):
The initial configuration will look similar to this screen shot. You will need to edit the fields as describe in the preceding steps before you can use the configuration. '''NOTE: be sure to replace the template user name settings "YOURUSERNAME" and "YOURUSERID" with the appropriate settings for your desktop and cluster account.'''
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
</pre>
# Copy the public key to cheaha (change USERID to your cheaha login id). You will be prompted for your cheaha password.
<pre>
$ ssh-copy-id -i $HOME/.ssh/id_rsa.pub USERID@cheaha.uabgrid.uab.edu
</pre>
# Run ssh-agent to load the key on your Linux workstation
<pre>
$ ssh-agent
Enter passphrase for ~/.ssh/id_rsa:  
Identity added: ~/.ssh/id_rsa (~/.ssh/id_rsa)
</pre>
# Now test that you can connect to cheaha without entering a password (again change USERID to your login name)
<pre>
$ ssh USERID@cheaha.uabgrid.uab.edu uptime


15:34:10 up 9 days,  5:11, 14 users,  load average: 0.25, 0.34, 0.50
[[Image:MATLAB_R2012a_configuration.png|none|x650px]]
</pre>
=== Mac ===


== Matlab from the Command Line ==
===== Validate the "cheaha" PCT Configuration =====


=== Matlab Versions ===
# Before starting validation please make sure the directory 'lustre/scratch/YOURUSERID/matlab' (please convert all settings to point to the new preferred location) or ''''/scratch/user/YOURUSERID/matlab'''' (preferred)  exists on the  scratch space on the Cheaha. If it does not please SSH into Cheaha and create the directory before proceeding.
Use the 'module' command to view a list of available Matlab versions. If the version that you require isn't listed, please open a help desk ticket to request the installation.
# Select Cheaha on the configuration manager page and click 'Start Validation'
# Wait for the validation to complete. This might take a few minutes and you ask for User credentials on Cheaha. All tests other than 'Matlabpool' validate on the Cheaha and the output is as shown.


The following is an example output of the command and doesn't necessarily represent the currently installed versions:
[[Image:Validation.png|none|x400px]]
<pre>
$ module avail mathworks


------------------------------------ /etc/modulefiles -------------------------------------
===== Begin Using MATLAB DCS from your Desktop =====
mathworks/R2009b
mathworks/R2009a


</pre>
The MATLAB DCS is now configured for Desktop usage. A simple parallel wave job "rParforWave" to test the configuration is described in [[MatLab_DCS_Examples]].
=== Simple Matlab Test ===
A simple test to verify that the Matlab client on Cheaha can check out a license from your server.


Set up your environment with the command:
A summary of the above steps is available at [[MATLAB_workshop_2011]] with additional examples and submit scripts available in the [[MATLAB_workshop_2011#Workshop_Demo.27s | workshop demo ]] section.
<pre>
$ module load mathworks/R2009b
</pre>
As a test, you can run MatLab and access your license server with
<pre>
$ matlab -c port@license-server -nodesktop -nojvm -r "rand, exit"
</pre>
For example:
<pre>
$ module load mathworks/R2009b
$ matlab -c 27000@licserver.uab.edu -nodesktop -nojvm -r "rand, exit"


                        < M A T L A B (R) >
=== MATLAB DCS from Cheaha ===
                Copyright 1984-2009 The MathWorks, Inc.
              Version 7.9.0.529 (R2009b) 64-bit (glnxa64)
                          August 12, 2009
  To get started, type one of these: helpwin, helpdesk, or demo.
  For product information, visit www.mathworks.com.


ans =
MATLAB can be started interactively from [[Cheaha]] via an SSH session using the [[MatLab CLI|command line]], X windows forwarding, or VNC.  This is very similar to using MATLAB from your desktop: in order to leverage the compute power of the cluster the Parallel Computing Toolbox must be configured to send tasks to the cluster scheduler.


    0.8147
If you do not follow these configuration steps, parallel tasks will be executed locally on the cluster log in node (head node). This will negatively impact your own and others interactive use of this log in node and may lead to your computations being stopped administratively.
</pre>
This will start matlab without a graphical display and without Java support.  This is good just to  
verify things work, but do not run any significant computations on the Cheaha head node!


MatLab computational work must be run on the compute nodes by submitting a job submission script to the SGE scheduler
==== MATLAB Submit Functions ====


=== Serial Matlab ===
The MATLAB submit functions create a cluster job context for your code. When running MATLAB from Cheaha, the submit functions are already installed and no additional actions are required by the user for this step.
Serial Matlab jobs have the following characteristics:
* Consumes one of your client licenses for the duration of the job
  * Does not use the distributed licenses available on cheaha
* Does not require the parallel computing toolbox
* Restricted to a single CPU core (slot)


See the next section for an example using the distributed computing license.
==== Parallel Computing Toolbox Configuration ====


Create a job script "matlabtest.qsub" making sure to change:
The Parallel Computing Toolbox (PCT) for your copy of MATLAB running in your cluster account must be configured to submit tasks to the compute nodes of the cluster.  Keep in mind that running MATLAB interactively on the cluster is the same as running it from your own desktop: MATLAB runs all tasks on the machine on which it is running unless it is told to send the work to another computer.  This condition holds even "inside" the Cheaha cluster: the cluster compute nodes are physically separate computers and MATLAB must request access via the scheduler just like any other job running on the cluster.
* YOUR_EMAIL_ADDRESS
* h_rt and s_rt to appropriate hard and soft runtime limits
* h_vmem to the maximum amount of memory that your job will use
<pre>
$ mkdir -p $UABGRID_SCRATCH/jobs/matlab/serial01/output
$ cd $UABGRID_SCRATCH/jobs/matlab/serial01
</pre>
<pre>
#!/bin/bash
#$ -S /bin/bash
#$ -cwd
#
#$ -N serialMatlab
#$ -l h_rt=00:10:00,s_rt=00:08:00,h_vmem=2G
#$ -j y
#
#$ -M YOUR_EMAIL_ADDRESS
#$ -m eas
#
module load mathworks/R2009b
#$ -V
matlab -c port@license-server -nodisplay -nojvm < matlab-script
</pre>
Then submit the script to the scheduler with
<pre>
$ cd $UABGRID_SCRATCH/jobs/matlab/serial01
$ qsub matlabtest.qsub
</pre>


Check on it with qstat.
Configuring the PCT when running MATLAB interactively from Cheaha is just like the configuration when running MATLAB from your desktop with two exceptions:
<pre>
* you must transfer any code or data to your Cheaha account explicitly outside of MATLAB using standard cluster procedures, aka SSH.
$ qstat -u $USER
* when MATLAB submits your tasks to the compute nodes, it benefits from the shared storage on the cluster and does not need to further copy your code and data to the compute nodes
</pre>


=== Distributed Matlab ===
To address these differences, follow the [[#Parallel Computing Toolbox Configuration|PCT instructions above]] and when [[#Personalize the "cheaha" PCT Configuration|editing the "cheaha" configuration]] modify the steps for the following fields:
These instructions provide an example of how to create and submit a distributed Matlab job on cheaha.
# '''Folder where job data is stored (DataLocation)''': specify a directory in your personal Cheaha account
# '''Function called when submitting parallel jobs (ParallelSubmitFnc)''': change the value to "{@ParallelSubmitFnc}"
# '''Function called when submitting distributed jobs (DistributedSubmitFnc)''': change the value to "{@DistributedSubmitFnc}"
# '''Job data location is accessible from both client and cluster nodes''': change this value to "True"


Distributed Matlab jobs use the following licenses:
Now save the "cheaha" configuring by clicking OK and proceeding to the [[#Validate the "cheaha" PCT Configuration|validation tests described above]].  Note: when running MATLAB interactively on Cheaha MatlabPool works and the validation tests are expected to pass (you'll see a green checkmark).
* Your own Matlab client license with the Parallel Computing Toolbox
* The Cheaha Distributed Computing license


The client license will only be needed for as long as it takes Matlab to start the job on the compute nodes (unless you keep the client open, for example using "waitForState(job)" in your Matlab script).
== References ==
* [http://www.mathworks.com/help/toolbox/distcomp/ Parallel Computing Toolbox on-line documentation]
* [http://www.mathworks.com/access/helpdesk/help/pdf_doc/distcomp/distcomp.pdf Parallel Computing Toolbox User's Guide (pdf)] - The 655 page MATLAB User Guide for the Parallel Computing Toolbox and is recommended reading!


The instructions are a work in progress, so please contact Research Computing support with any questions or corrections.
{{MATLAB Support}}


First, create the working directory for the job
[[Category:MATLAB]][[Category:MATLAB installation]]
<pre>
$ mkdir -p $UABGRID_SCRATCH/jobs/matlab/distrib01/output
$ cd $UABGRID_SCRATCH/jobs/matlab/distrib01
 
</pre>
 
Next, create a simple 2 task distributed Matlab script called "distrib.m" make sure to change:
* email to your email address
* s_rt to an appropriate soft wall time limit
* h_rt to the maximum wall time for your job
* mem_free to the maximum memory needed for each task
* outputDirectory to the directory where results should be stored
 
Don't make any changes to the section labeled "Configure the scheduler"
<pre>
% Always set these variables
email          = 'YOUR_EMAIL_ADDRESS';
s_rt            = '00:05:00';
h_rt            = '00:07:00';
mem_free        = '1G';
clusterHost    = 'cheaha.uabgrid.uab.edu';
scratch        = getenv('UABGRID_SCRATCH');
outputDirectory  = [scratch, '/jobs/matlab/distrib01/output'];
 
% Configure the scheduler
sched = findResource('scheduler', 'type', 'generic');
set(sched, 'DataLocation'      , outputDirectory);
set(sched, 'ClusterMatlabRoot', '/share/apps/mathworks/R2009b');
set(sched, 'HasSharedFilesystem',  true              );
set(sched, 'ClusterOsType'      , 'unix'            );
set(sched, 'SubmitFcn', {@sgeSubmitFcn, h_rt, s_rt, mem_free, email});
set(sched, 'DestroyJobFcn', {@sgeDestroyJob});
set(sched, 'GetJobStateFcn', {@sgeGetJobState});
get(sched)
job = createJob(sched);
 
% start of user specific commands
createTask(job, @rand, 1, {3,3});
createTask(job, @rand, 1, {3,3});
 
submit(job)
</pre>
 
Running the Matlab script will submit 2 SGE single slot (CPU) jobs, one for each task. The Parallel Computing Toolbox requires Java VM, so notice that for this job we do not include the "-nojvm" switch!
<pre>
$ module load mathworks/R2009b
$ matlab -c port@license-server -nodisplay < distrib.m
</pre>
 
Check qstat to see that the scheduler now has 2 jobs running, one for each task
<pre>
$ qstat -u $USER
 
job-ID  prior  name      user        state submit/start at    queue                          slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
110839 0.50167 Job1.1    jdoe        r    03/10/2010 16:32:37 all.q@compute-0-12.local          1
110840 0.50083 Job1.2    jdoe        r    03/10/2010 16:32:37 all.q@compute-0-12.local          1
</pre>
 
The job output can be found in the "output" directory
 
=== Parallel Matlab ===
These instructions provide an example of how to create and submit a parallel Matlab job on cheaha. Parallel Matlab jobs require two separate licenses:
* Your own client license that includes the Parallel Computing Toolbox
* The Cheaha 128 node Distributed Computing license
 
The client license will only be needed for as long as it takes Matlab to start the job on the compute nodes (unless you keep the client open, for example using "waitForState(job)" in your Matlab script).
 
Check out this [http://www.mathworks.com/access/helpdesk/help/toolbox/distcomp/brjw1e5-1.html Matlab Help Page] for a quick overview of using parallel code in your Matlab scripts.
 
First, create the working directory for the job
<pre>
$ mkdir -p $UABGRID_SCRATCH/jobs/matlab/paralle01/output
$ cd $UABGRID_SCRATCH/jobs/matlab/parallel01
 
</pre>
 
Next, create a simple 4 slot parallel Matlab script called "parjob.m" make sure to change:
* email to your email address
* s_rt to an appropriate soft wall time limit
* h_rt to the maximum wall time for your job
* mem_free to the maximum memory needed for each task
* outputDirectory to the directory where results should be stored
 
Don't make any changes to the section labeled "Configure the scheduler"
<pre>
% Always set these variables
email          = 'YOUR_EMAIL_ADDRESS';
s_rt            = '00:05:00';
h_rt            = '00:07:00';
mem_free        = '2G';
clusterHost    = 'cheaha.uabgrid.uab.edu';
scratch        = getenv('UABGRID_SCRATCH');
outputDirectory = [scratch, '/jobs/matlab/parallel01/output'];
 
% Configure the scheduler
sched = findResource('scheduler', 'type', 'generic');
set(sched, 'DataLocation'      , outputDirectory);
set(sched, 'ClusterMatlabRoot', '/share/apps/mathworks/R2009b');
set(sched, 'HasSharedFilesystem',  true              );
set(sched, 'ClusterOsType'      , 'unix'            );
set(sched, 'ParallelSubmitFcn', {@sgeParallelSubmitFcn, h_rt, s_rt, mem_free, email});
set(sched, 'DestroyJobFcn', {@sgeDestroyJob});
set(sched, 'GetJobStateFcn', {@sgeGetJobState});
get(sched)
pjob = createParallelJob(sched);
 
% start of user specific commands
createTask(pjob, 'rand', 1, {4});
set(pjob, 'MinimumNumberOfWorkers', 4);
set(pjob, 'MaximumNumberOfWorkers', 4);
 
submit(pjob)
</pre>
 
Running the Matlab script will submit 1 SGE job requesting 4 slots (cpu cores). The Parallel Computing Toolbox requires Java VM, so notice that for this job we do not include the "-nojvm" switch!
<pre>
$ module load mathworks/R2009b
$ matlab -c port@license-server -nodisplay < parjob.m
</pre>
 
Check qstat to see that the scheduler now has 2 jobs running, one for each task
<pre>
$ qstat -u $USER
 
job-ID  prior  name   user        state submit/start at    queue                          slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
110857 0.00000 Job1   jdoe        r    03/10/2010 17:20:08                                    4
 
</pre>
 
The job output can be found in the "output" directory
 
=== ParFor Parallel Example ===
This example will utilize the parfor parallel loop as [http://www.mathworks.com/products/parallel-computing/parallel/accelerate.html defined here].

Latest revision as of 19:41, 11 February 2019

Ambox important.png

This page is Deprecated
To Use MATLAB with the SLURM scheduler on Cheaha please click the link below

MATLAB with SLURM


The MATLAB Distributed Computing Server (MATLAB DCS) is a parallel computing extension to MATLAB that enables processing to be spread across a large number of worker nodes, accelerating the speed at which compute intensive operations can complete.

UAB IT Research Computing maintains a 128 worker node license for the the Cheaha computing platform. In order to use DCS on Cheaha, you will need to use a MATLAB instance with the Parallel Computing Toolbox installed.

In order to leverage the MATLAB worker nodes on Cheaha the Parallel Computing Toolbox will need to be configured to submit compute tasks to Cheaha by following the steps in this document.

Please see the MATLAB application page for more information and a general overview of MATLAB and its use at UAB.

Overview

The following outline highlights the steps involved to configure your MatLab install and write programs that submit tasks to the worker nodes of the Distributed Computing Server on Cheaha:

  • Configure the Task Submit Environment (One-time Setup)
    • Install MatLab with the Parallel Computing Toolbox on your Windows / Linux / Mac workstation
    • Download and extract the MatLab task submission functions to your workstation MatLab environment
    • Define the "cheaha" parallel configuration in your workstation MatLab environment to submit tasks to Cheaha
    • Run the validation tests to ensure your "cheaha" parallel configuration works
  • Develop and Run Parallel Computing Applications
    • Write, test and debug your parallel code on your local workstation using the default "local" parallel configuration
    • Once your code works, select the "cheaha" parallel configuration to submit tasks to the Cheaha cluster. Note: your workstation MatLab application does not need to keep running after the tasks are submitted.
    • You will receive an email when the tasks you submitted are complete
    • Use your workstation MatLab application to retrieve the results
    • When you are finished with your job contexts, clean up the job related content to free disk space

Using MATLAB DCS

The MATLAB Distributed Computing Services (DCS) are accessed via the Parallel Computing Toolbox (PCT) which is installed as part of your desktop MATLAB installation. The PCT allows MATLAB running on your workstation to send MATLAB code and data (tasks) to the cluster directly from the comfort of your familiar MATLAB environment on your desktop. This makes the expanded compute power of Cheaha available to processes work loads that exceed the capabilities of your desktop computer. Once your tasks are submitted to Cheaha, your desktop MATLAB is also free to move on to other tasks or be closed completely, freeing your desktop or laptop for your other activities.

Configuring the Parallel Computing Toolbox involves three steps documented below:

  1. install MATLAB submit functions on your workstation
  2. configure the "cheaha" parallel computing target to which PCT tasks can be submitted
  3. run the validation tests to confirm a working installation.

This page documents the DCS configuration for MATLAB 2010b and later. For DCS configuration instructions on previous versions of MATLAB, please see the page MatLab DCS R2010a and Earlier

Using MATLAB DCS requires you have a cluster account on Cheaha. Please request an account by sending an email to [[1]] and include your campus affiliation and a brief statement of your research interests for using the cluster.

MATLAB DCS from Your Desktop

MATLAB Submit Functions

The MATLAB submit functions create a cluster job context for your code and are responsible for transferring your code and the data it analyzes to the cluster for processing.

These submit functions must be installed on your computer and must be accessible to MATLAB via the MATLAB PATH environment. The easiest way to accomplish this is to copy the submit functions to the default directory created for by MATLAB. These directories on the respective operating systems are listed below.

All operating systems (Windows, Linux and Mac) are supported by the same set of submit functions. The functions are written in MATLAB making them cross-platform and only dependent on the version of MATLAB in use.

  1. Download the MATLAB submit functions
  2. Unzip the files to a directory included in your MATLAB PATH setting. Recommended locations are:
    • Windows: 
      My Documents\MATLAB
    • Linux: 
      $HOME/Documents/MATLAB
    • Mac: 
      $HOME/Documents/MATLAB

Once the submit function files have been downloaded and unzipped in the above paths, restart MATLAB to ensure they are properly loaded in your environment.

NOTE: If you choose not to use the above path recommendations, your MATLAB PATH may be viewed/altered by starting the MATLAB client on your workstation and clicking File -> Set Path and adding the path in which you unpacked the submit functions.

Parallel Computing Toolbox Configuration

The Parallel Computing Toolbox (PCT) enables language extensions in MATLAB that support dividing your application into tasks that can be executed in parallel. By default, all of these tasks will run on your local workstation using the pre-defined "local" PCT configuration.

To run these tasks on the Cheaha compute cluster, a new configuration for the PCT must be defined. In this section we will create the "cheaha" configuration and run a quick validation test to confirm its operation.

Prior to continuing, make sure you:

  • can establish an SSH connection to Cheaha
  • have followed the steps in the previous section
Create the "cheaha" PCT Configuration

Download and save the Cheaha cluster configuration file for your MATLAB version

  1. R2010b, R2011a, R2011b cluster configuration file
    • Start MATLAB on your workstation
    • Click the "Parallel" menu
    • Click "Manage Configurations"
    • In the "Configurations Manager" window, click "File -> Import"
    • Browse to the location where you saved the cheaha-R2011b.mat file, select it, and click "Open"
  2. R2012a cluster configuration file
    • Start MATLAB R2012a on your workstation
    • Click the "Parallel" menu
    • Click "Manage Cluster Profiles"
    • In the "Cluster Profile Manager" window, click the "Import" button on the toolbar
    • Browse to the location where you saved the cheaha-R2012a.settings file, select it, and click "Open"

The Configuration Manager for R2011b and prior should now list a new entry named "cheaha" as shown in the following image, R2012a and later will also have a new entry in the Cluster Profile list:

2011 config mngr.png
Personalize the "cheaha" PCT Configuration -2011b and earlier
  1. Double click on cheaha in the Configuration Manager window to open the configuration editor. (Note: stretch the "Generic Scheduler Configuration Properties" window to the right so that you can view all of the text in the fields making it easier to read and edit correctly.)
  2. Edit the following fields to use your personal data directories
    • ClusterMatlabRoot: Make sure that the Root directory of MATLAB installation for workers matches the exact version of MATLAB you are using on your workstation. In this example /share/apps/mathworks/R2011a matches a MATLAB R2011a workstation install. Change the "R2011a" to match your workstation MATLAB version.
    • DataLocation  : Change the directory path where job data is stored to an existing directory on your workstation where MATLAB can stage job files.
    • ParallelSubmitFcn: Change the text "YOURUSERID" to your login id on Cheaha
    • SubmitFcn  : Change the text "YOURUSERID" to your login id on Cheaha
  3. Click 'OK'to save the configuration
  4. SSH to cheaha and make sure to create the $USER_SCRATCH/matlab directory. If this directory does not exist, the parallel computing toolbox jobs will fail.

The initial configuration will look similar to this screen shot. You will need to edit the fields as describe in the preceding steps before you can use the configuration. NOTE: be sure to replace the template user name settings "YOURUSERNAME" with the appropriate settings for your desktop and cluster account.

Cheaha parallel config.png
Personalize the "cheaha" PCT Configuration -2012a
  1. Double click on cheaha in the Configuration Manager window to open the configuration editor. (Note: stretch the "Generic Scheduler Configuration Properties" window to the right so that you can view all of the text in the fields making it easier to read and edit correctly.)
  2. Edit the following fields to use your personal data directories
    • ClusterMatlabRoot: Make sure that the Root directory of MATLAB installation for workers matches the exact version of MATLAB you are using on your workstation. In this example /share/apps/mathworks/R2012a matches a MATLAB R2012a workstation install. Change the "R2012a" to match your workstation MATLAB version.
    • DataLocation  : Change the directory path where job data is stored to an existing directory on your workstation where MATLAB can stage job files.
    • independentSubmitFcn: Change the text "YOURUSERID" to your login id on Cheaha
    • communicatingSubmitFcn  : Change the text "YOURUSERID" to your login id on Cheaha
  3. Click 'OK'to save the configuration
  4. SSH to cheaha and make sure to create the $USER_SCRATCH/matlab directory. If this directory does not exist, the parallel computing toolbox jobs will fail.

The initial configuration will look similar to this screen shot. You will need to edit the fields as describe in the preceding steps before you can use the configuration. NOTE: be sure to replace the template user name settings "YOURUSERNAME" and "YOURUSERID" with the appropriate settings for your desktop and cluster account.

MATLAB R2012a configuration.png
Validate the "cheaha" PCT Configuration
  1. Before starting validation please make sure the directory 'lustre/scratch/YOURUSERID/matlab' (please convert all settings to point to the new preferred location) or '/scratch/user/YOURUSERID/matlab' (preferred) exists on the scratch space on the Cheaha. If it does not please SSH into Cheaha and create the directory before proceeding.
  2. Select Cheaha on the configuration manager page and click 'Start Validation'
  3. Wait for the validation to complete. This might take a few minutes and you ask for User credentials on Cheaha. All tests other than 'Matlabpool' validate on the Cheaha and the output is as shown.
Validation.png
Begin Using MATLAB DCS from your Desktop

The MATLAB DCS is now configured for Desktop usage. A simple parallel wave job "rParforWave" to test the configuration is described in MatLab_DCS_Examples.

A summary of the above steps is available at MATLAB_workshop_2011 with additional examples and submit scripts available in the workshop demo section.

MATLAB DCS from Cheaha

MATLAB can be started interactively from Cheaha via an SSH session using the command line, X windows forwarding, or VNC. This is very similar to using MATLAB from your desktop: in order to leverage the compute power of the cluster the Parallel Computing Toolbox must be configured to send tasks to the cluster scheduler.

If you do not follow these configuration steps, parallel tasks will be executed locally on the cluster log in node (head node). This will negatively impact your own and others interactive use of this log in node and may lead to your computations being stopped administratively.

MATLAB Submit Functions

The MATLAB submit functions create a cluster job context for your code. When running MATLAB from Cheaha, the submit functions are already installed and no additional actions are required by the user for this step.

Parallel Computing Toolbox Configuration

The Parallel Computing Toolbox (PCT) for your copy of MATLAB running in your cluster account must be configured to submit tasks to the compute nodes of the cluster. Keep in mind that running MATLAB interactively on the cluster is the same as running it from your own desktop: MATLAB runs all tasks on the machine on which it is running unless it is told to send the work to another computer. This condition holds even "inside" the Cheaha cluster: the cluster compute nodes are physically separate computers and MATLAB must request access via the scheduler just like any other job running on the cluster.

Configuring the PCT when running MATLAB interactively from Cheaha is just like the configuration when running MATLAB from your desktop with two exceptions:

  • you must transfer any code or data to your Cheaha account explicitly outside of MATLAB using standard cluster procedures, aka SSH.
  • when MATLAB submits your tasks to the compute nodes, it benefits from the shared storage on the cluster and does not need to further copy your code and data to the compute nodes

To address these differences, follow the PCT instructions above and when editing the "cheaha" configuration modify the steps for the following fields:

  1. Folder where job data is stored (DataLocation): specify a directory in your personal Cheaha account
  2. Function called when submitting parallel jobs (ParallelSubmitFnc): change the value to "{@ParallelSubmitFnc}"
  3. Function called when submitting distributed jobs (DistributedSubmitFnc): change the value to "{@DistributedSubmitFnc}"
  4. Job data location is accessible from both client and cluster nodes: change this value to "True"

Now save the "cheaha" configuring by clicking OK and proceeding to the validation tests described above. Note: when running MATLAB interactively on Cheaha MatlabPool works and the validation tests are expected to pass (you'll see a green checkmark).

References

MATLAB Support / Mailing List

As with any application or computer language, learning to use MATLAB to analyze data or to develop or modify MATLAB applications is an individual responsibility. There is ample application documentation available from the Mathworks website, potential outreach to colleagues who also use MATLAB, and options for consultation with Mathworks. Mathworks also host on-campus training seminars several times a year and provides many on-line learning tutorials.

Installation support for MATLAB at UAB is provided by your local IT support organization and the Docs wiki.

Mathworks Website

Your first and best option for application-specific questions on MATLAB is to refer to the on-line MATLAB documentation. The Mathworks site also provides a a support matrix and an on-line knowledge base.

UAB MATLAB Wiki

The MATLAB page on the Docs wiki is the starting point for installing MATLAB at UAB and, optionally, configuring it to use cluster computing. All users are encouraged to contribute to the MATLAB knowledge in this wiki, especially if you see areas where improvements are needed. Remember, this knowledge base is only as good as the people who contribute to it.

Contributing to the wiki is as easy as clicking the login link on the top-right of the page and signing in with your UAB BlazerID. If you are unsure about making an edit, you can make suggestions for improvement on the page's Discussion tab or discuss the proposed improvement in the MATLAB user group.

UAB MATLAB User Group

At UAB, MATLAB installation support is provided by your local IT support group. Support for application specific questions is available from peers in your research group. We realize that some people are not as familiar with MATLAB as others. For this reason, we have established a MATLAB user forum (mailing list) where users of MATLAB at UAB can help answer each others questions.

This is a network of volunteers sharing their knowledge with peers. You are encouraged to reach out to this community for questions on using MATLAB by

Archives of MATLAB user group discussions are available on-line at https://vo.uabgrid.uab.edu/sympa/arc/matlab-user. You may find your question is already answered in these archives.


UAB MATLAB announce mailing list

To receive information about UAB's MATLAB license and announcements please subscribe to the matlab-annc mailing list by

Retrieved from "https://docs.uabgrid.uab.edu/w/index.php?title=MATLAB_DCS&oldid=5891"