Cheaha - User contributions [en]

Cheaha GettingStarted

2017-10-15T22:10:45Z

Puri@uab.edu: /* Hardware */

Cheaha is a cluster computing environment for UAB researchers. Information about the history and future plans for Cheaha is available on the [[Cheaha]] page.

== Access (Cluster Account Request) ==

To request an account on [[Cheaha]], please {{CheahaAccountRequest}}. Please include some background information about the work you plan on doing on the cluster and the group you work with, ie. your lab or affiliation.

Usage of Cheaha is governed by [http://www.uabgrid.uab.edu/aup UAB's Acceptable Use Policy (AUP)] for computer resources.

The official DNS name of Cheaha's frontend machine is ''cheaha.rc.uab.edu''. If you want to refer to the machine as ''cheaha'', you'll have to either add the "rc.uab.edu" to you computer's DNS search path. On Unix-derived systems (Linux, Mac) you can edit your computers /etc/resolv.conf as follows (you'll need administrator access to edit this file)
<pre>
search rc.uab.edu
</pre>
Or you can customize your SSH configuration to use the short name "cheaha" as a connection name. On systems using OpenSSH you can add the following to your ~/.ssh/config file

<pre>
Host cheaha
Hostname cheaha.rc.uab.edu
</pre>

== Login ==
===Overview===
Once your account has been created, you'll receive an email containing your user ID, generally your Blazer ID. Logging into Cheaha requires an SSH client. Most UAB Windows workstations already have an SSH client installed, possibly named '''SSH Secure Shell Client''' or [http://www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY]. Linux and Mac OS X systems should have an SSH client installed by default.

Usage of Cheaha is governed by [http://www.uabgrid.uab.edu/aup UAB's Acceptable Use Policy (AUP)] for computer resources.

===Client Configuration===
This section will cover steps to configure Windows, Linux and Mac OS X clients to connect to Cheaha.
====Linux====
Linux systems, regardless of the flavor (RedHat, SuSE, Ubuntu, etc...), should already have an SSH client on the system as part of the default install.
# Start a terminal (on RedHat click Applications -> Accessories -> Terminal, on Ubuntu Ctrl+Alt+T)
# At the prompt, enter the following command to connect to Cheaha ('''Replace blazerid with your Cheaha userid''')
ssh '''blazerid'''@cheaha.rc.uab.edu

====Mac OS X====
Mac OS X is a Unix operating system (BSD) and has a built in ssh client.
# Start a terminal (click Finder, type Terminal and double click on Terminal under the Applications category)
# At the prompt, enter the following command to connect to Cheaha ('''Replace blazerid with your Cheaha userid''')
ssh '''blazerid'''@cheaha.rc.uab.edu

====Windows====
There are many SSH clients available for Windows, some commercial and some that are free (GPL). This section will cover two clients that are commonly found on UAB Windows systems.
=====MobaXterm=====
[http://mobaxterm.mobatek.net/ MobaXterm] is a free (also available for a price in a Profession version) suite of SSH tools. Of the Windows clients we've used, MobaXterm is the easiest to use and feature complete. [http://mobaxterm.mobatek.net/features.html Features] include (but not limited to):
* SSH client (in a handy web browser like tabbed interface)
* Embedded Cygwin (which allows Windows users to run many Linux commands like grep, rsync, sed)
* Remote file system browser (graphical SFTP)
* X11 forwarding for remotely displaying graphical content from Cheaha
* Installs without requiring Windows Administrator rights

Start MobaXterm and click the Session toolbar button (top left). Click SSH for the session type, enter the following information and click OK. Once finished, double click cheaha.rc.uab.edu in the list of Saved sessions under PuTTY sessions:
{| border="1" cellpadding="5"
!Field
!Cheaha Settings
|-
|'''Remote host'''
|cheaha.rc.uab.edu
|-
|'''Port'''
|22
|-
|}

=====PuTTY=====
[http://www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY] is a free suite of SSH and telnet tools written and maintained by [http://www.pobox.com/~anakin/ Simon Tatham]. PuTTY supports SSH, secure FTP (SFTP), and X forwarding (XTERM) among other tools.

* Start PuTTY (Click START -> All Programs -> PuTTY -> PuTTY). The 'PuTTY Configuration' window will open
* Use these settings for each of the clusters that you would like to configure
{| border="1" cellpadding="5"
!Field
!Cheaha Settings
|-
|'''Host Name (or IP address)'''
|cheaha.rc.uab.edu
|-
|'''Port'''
|22
|-
|'''Protocol'''
|SSH
|-
|'''Saved Sessions'''
|cheaha.rc.uab.edu
|-
|}
* Click '''Save''' to save the configuration, repeat the previous steps for the other clusters
* The next time you start PuTTY, simply double click on the cluster name under the 'Saved Sessions' list

=====SSH Secure Shell Client=====
SSH Secure Shell is a commercial application that is installed on many Windows workstations on campus and can be configured as follows:
* Start the program (Click START -> All Programs -> SSH Secure Shell -> Secure Shell Client). The 'default - SSH Secure Shell' window will open
* Click File -> Profiles -> Add Profile to open the 'Add Profile' window
* Type in the name of the cluster (for example: cheaha) in the field and click 'Add to Profiles'
* Click File -> Profiles -> Edit Profiles to open the 'Profiles' window
* Single click on your new profile name
* Use these settings for the clusters
{| border="1" cellpadding="5"
!Field
!Cheaha Settings
|-
|'''Host name'''
|cheaha.rc.uab.edu
|-
|'''User name'''
|blazerid (insert your blazerid here)
|-
|'''Port'''
|22
|-
|'''Protocol'''
|SSH
|-
|'''Encryption algorithm'''
|<Default>
|-
|'''MAC algorithm'''
|<Default>
|-
|'''Compression'''
|<None>
|-
|'''Terminal answerback'''
|vt100
|-
|}
* Leave 'Connect through firewall' and 'Request tunnels only' unchecked
* Click '''OK''' to save the configuration, repeat the previous steps for the other clusters
* The next time you start SSH Secure Shell, click 'Profiles' and click the cluster name

=== Logging in to Cheaha ===
No matter which client you use to connect to the Cheaha, the first time you connect, the SSH client should display a message asking if you would like to import the hosts public key. Answer '''Yes''' to this question.

* Connect to Cheaha using one of the methods listed above
* Answer '''Yes''' to import the cluster's public key
** Enter your BlazerID password

* After successfully logging in for the first time, You may see the following message '''just press ENTER for the next three prompts, don't type any passphrases!'''

It doesn't appear that you have set up your ssh key.
This process will make the files:
/home/joeuser/.ssh/id_rsa.pub
/home/joeuser/.ssh/id_rsa
/home/joeuser/.ssh/authorized_keys

Generating public/private rsa key pair.
Enter file in which to save the key (/home/joeuser/.ssh/id_rsa):
** Enter file in which to save the key (/home/joeuser/.ssh/id_rsa):'''Press Enter'''
** Enter passphrase (empty for no passphrase):'''Press Enter'''
** Enter same passphrase again:'''Press Enter'''
Your identification has been saved in /home/joeuser/.ssh/id_rsa.
Your public key has been saved in /home/joeuser/.ssh/id_rsa.pub.
The key fingerprint is:
f6:xx:xx:xx:xx:dd:9a:79:7b:83:xx:f9:d7:a7:d6:27 joeuser@cheaha.rc.uab.edu

==== Users without a blazerid (collaborators from other universities) ====
** If you were issued a temporary password, enter it (Passwords are CaSE SensitivE!!!) You should see a message similar to this
You are required to change your password immediately (password aged)

WARNING: Your password has expired.
You must change your password now and login again!
Changing password for user joeuser.
Changing password for joeuser
(current) UNIX password:
*** (current) UNIX password: '''Enter your temporary password at this prompt and press enter'''
*** New UNIX password: '''Enter your new strong password and press enter'''
*** Retype new UNIX password: '''Enter your new strong password again and press enter'''
*** After you enter your new password for the second time and press enter, the shell may exit automatically. If it doesn't, type exit and press enter
*** Log in again, this time use your new password

Congratulations, you should now have a command prompt and be ready to start [[Cheaha_GettingStarted#Example_Batch_Job_Script | submitting jobs]]!!!

== Hardware ==
[[Image:Chehah2_2016.png|center|thumb|450px|Logical Diagram of Cheaha Configuration]]

The Cheaha Compute Platform includes commodity compute hardware, totaling 2800 compute cores and over 4.7PB of usable storage (6.6PB raw capacity). The following descriptions highlight the current hardware profile that provides an aggregate theoretical peak performance of 468 teraflops.

* Compute
** 36 Compute Nodes with two 12 core processors (Intel Xeon E5-2680 v3 2.5GHz) with 128GB DDR4 RAM, FDR InfiniBand and 10GigE network cards
** 38 Compute Nodes with two 12 core processors (Intel Xeon E5-2680 v3 2.5GHz) with 256GB DDR4 RAM, FDR InfiniBand and 10GigE network cards
** 14 Compute Nodes with two 12 core processors (Intel Xeon E5-2680 v3 2.5GHz) with 384GB DDR4 RAM, FDR InfiniBand and 10GigE network cards
** 4 Compute Nodes with Nvidia Tesla K80 and two 12 core processors (Intel Xeon E5-2680 v3 2.5GHz) with 128GB DDR4 RAM, FDR InfiniBand and 10GigE network cards
** 4 Compute Nodes with Intel Phi coprocessor SE10/7120 and two 12 core processors (Intel Xeon E5-2680 v3 2.5GHz) with 128GB DDR4 RAM, FDR InfiniBand and 10GigE network cards
** 18 Compute Nodes with two 14 core processors (Intel Xeon E5-2680 v4 2.4GHz)with 256GB DDR4 RAM, four NVIDIA Tesla P100 16GB GPUs, EDR InfiniBand and 10GigE network cards

* Networking
**FDR and EDR InfiniBand Switch
** 10Gigabit Ethernet Switch

* Storage -- DDN SFA12KX with GPFS)
** 2 x 12KX40D-56IB controllers
** 10 x SS8460 disk enclosures
** 825 x 4K SAS drives

* Management
** Management node and gigabit switch for cluster management
** Bright Advanced Cluster Management software licenses

== Cluster Software ==
* BrightCM 7.2
* CentOS 7.2 x86_64
* [[Slurm]] 15.08

== Queuing System ==
All work on Cheaha must be submitted to '''our queuing system ([[Slurm]])'''. A common mistake made by new users is to run 'jobs' on the login node. This section gives a basic overview of what a queuing system is and why we use it.
=== What is a queuing system? ===
* Software that gives users fair allocation of the cluster's resources
* Schedules jobs based using resource requests (the following are commonly requested resources, there are many more that are available)
** Number of processors (often referred to as "slots")
** Maximum memory (RAM) required per slot
** Maximum run time
* Common queuing systems:
** '''[[Slurm]]'''
** Sun Grid Engine (Also know as SGE, OGE, GE)
** OpenPBS
** Torque
** LSF (load sharing facility)

[http://slurm.schedmd.com/ Slurm] is a queue management system and stands for Simple Linux Utility for Resource Management. Slurm was developed at the Lawrence Livermore National Lab and currently runs some of the largest compute clusters in the world. '''[[Slurm]]''' is now the primary job manager on Cheaha, it replaces SUN Grid Engine ([[https://docs.uabgrid.uab.edu/wiki/Cheaha_GettingStarted_deprecated SGE]]) the job manager used earlier. Instructions of using SLURM and writing SLURM scripts for jobs submission on Cheaha can be found '''[[Slurm | here]]'''.

=== Typical Workflow ===
* Stage data to $USER_SCRATCH (your scratch directory)
* Research how to run your code in "batch" mode. Batch mode typically means the ability to run it from the command line without requiring any interaction from the user.
* Identify the appropriate resources needed to run the job. The following are mandatory resource requests for all jobs on Cheaha
** Maximum memory (RAM) required per slot
** Maximum runtime
* Write a job script specifying queuing system parameters, resource requests and commands to run program
* Submit script to queuing system (sbatch script.job)
* Monitor job (squeue)
* Review the results and resubmit as necessary
* Clean up the scratch directory by moving or deleting the data off of the cluster

=== Resource Requests ===
Accurate resource requests are extremely important to the health of the over all cluster. In order for Cheaha to operate properly, the queing system must know how much runtime and RAM each job will need.

==== Mandatory Resource Requests ====

* -t, --time=<time>
Set a limit on the total run time of the job allocation. If the requested time limit exceeds the partition's time limit, the job will be left in a PENDING state (possibly indefinitely).
* For Array jobs, this represents the maximum run time for each task
** For serial or parallel jobs, this represents the maximum run time for the entire job

* --mem-per-cpu=<MB>
Mimimum memory required per allocated CPU in MegaBytes.

==== Other Common Resource Requests ====
* -N, --nodes=<minnodes[-maxnodes]>
Request that a minimum of minnodes nodes be allocated to this job. A maximum node count may also be specified with maxnodes. If only one number is specified, this is used as both the minimum and maximum node count.

* -n, --ntasks=<number>
sbatch does not launch tasks, it requests an allocation of resources and submits a batch script. This option advises the Slurm controller that job steps run within the allocation will launch a maximum of number tasks and to provide for sufficient resources. The default is one task per node

* --mem=<MB>
Specify the real memory required per node in MegaBytes.

* -c, --cpus-per-task=<ncpus>
Advise the Slurm controller that ensuing job steps will require ncpus number of processors per task. Without this option, the controller will just try to allocate one processor per task.

* -p, --partition=<partition_names>
Request a specific partition for the resource allocation. Available partitions are: express(max 2 hrs), short(max 12 hrs), medium(max 50 hrs), long(max 150 hrs), sinteractive(0-2 hrs)

=== Submitting Jobs ===
Batch Jobs are submitted on Cheaha by using the "sbatch" command. The full manual for sbtach is available by running the following command
man sbatch

==== Job Script File Format ====
To submit a job to the queuing systems, you will first define your job in a script (a text file) and then submit that script to the queuing system.

The script file needs to be '''formatted as a UNIX file''', not a Windows or Mac text file. In geek speak, this means that the end of line (EOL) character should be a line feed (LF) rather than a carriage return line feed (CRLF) for Windows or carriage return (CR) for Mac.

If you submit a job script formatted as a Windows or Mac text file, your job will likely fail with misleading messages, for example that the path specified does not exist.

Windows '''Notepad''' does not have the ability to save files using the UNIX file format. Do NOT use Notepad to create files intended for use on the clusters. Instead use one of the alternative text editors listed in the following section.

===== Converting Files to UNIX Format =====
====== Dos2Unix Method ======
The lines below that begin with $ are commands, the $ represents the command prompt and should not be typed!

The dos2unix program can be used to convert Windows text files to UNIX files with a simple command. After you have copied the file to your home directory on the cluster, you can identify that the file is a Windows file by executing the following (Windows uses CR LF as the line terminator, where UNIX uses only LF and Mac uses only CR):
<pre>
$ file testfile.txt

testfile.txt: ASCII text, with CRLF line terminators
</pre>

Now, convert the file to UNIX
<pre>
$ dos2unix testfile.txt

dos2unix: converting file testfile.txt to UNIX format ...
</pre>

Verify the conversion using the file command
<pre>
$ file testfile.txt

testfile.txt: ASCII text
</pre>

====== Alternative Windows Text Editors ======
There are many good text editors available for Windows that have the capability to save files using the UNIX file format. Here are a few:
* [[http://www.geany.org/ Geany]] is an excellent free text editor for Windows and Linux that supports Windows, UNIX and Mac file formats, syntax highlighting and many programming features. To convert from Windows to UNIX click '''Document''' click '''Set Line Endings''' and then '''Convert and Set to LF (Unix)'''
* [[http://notepad-plus.sourceforge.net/uk/site.htm Notepad++]] is a great free Windows text editor that supports Windows, UNIX and Mac file formats, syntax highlighting and many programming features. To convert from Windows to UNIX click '''Format''' and then click '''Convert to UNIX Format'''
* [[http://www.textpad.com/ TextPad]] is another excellent Windows text editor. TextPad is not free, however.

==== Example Batch Job Script ====
A shared cluster environment like Cheaha uses a job scheduler to run tasks on the cluster to provide optimal resource sharing among users. Cheaha uses a job scheduling system call Slurm to schedule and manage jobs. A user needs to tell Slurm about resource requirements (e.g. CPU, memory) so that it can schedule jobs effectively. These resource requirements along with actual application code can be specified in a single file commonly referred as 'Job Script/File'. Following is a simple job script that prints job number and hostname.

'''Note:'''Jobs '''must request''' the appropriate partition (ex: ''--partition=short'') to satisfy the jobs resource request (maximum runtime, number of compute nodes, etc...)
<pre>
#!/bin/bash
#
#SBATCH --job-name=test
#SBATCH --output=res.txt
#SBATCH --ntasks=1
#SBATCH --partition=express
#SBATCH --time=10:00
#SBATCH --mem-per-cpu=100
#SBATCH --mail-type=FAIL
#SBATCH --mail-user=YOUR_EMAIL_ADDRESS

srun hostname
srun sleep 60
</pre>

Lines starting with '#SBATCH' have a special meaning in the Slurm world. Slurm specific configuration options are specified after the '#SBATCH' characters. Above configuration options are useful for most job scripts and for additional configuration options refer to Slurm commands manual. A job script is submitted to the cluster using Slurm specific commands. There are many commands available, but following three commands are the most common:
* sbatch - to submit job
* scancel - to delete job
* squeue - to view job status

We can submit above job script using sbatch command:
<pre>
$ sbatch HelloCheaha.sh
Submitted batch job 52707
</pre>

When the job script is submitted, Slurm queues it up and assigns it a job number (e.g. 52707 in above example). The job number is available inside job script using environment variable $JOB_ID. This variable can be used inside job script to create job related directory structure or file names.

=== Interactive Resources ===
Login Node (the host that you connected to when you setup the SSH connection to Cheaha) is supposed to be used for submitting jobs and/or lighter prep work required for the job scripts. '''Do not run heavy computations on the login node'''. If you have a heavier workload to prepare for a batch job (eg. compiling code or other manipulations of data) or your compute application requires interactive control, you should request a dedicated interactive node for this work.

Interactive resources are requested by submitting an "interactive" job to the scheduler. Interactive jobs will provide you a command line on a compute resource that you can use just like you would the command line on the login node. The difference is that the scheduler has dedicated the requested resources to your job and you can run your interactive commands without having to worry about impacting other users on the login node.

Interactive jobs, that can be run on command line, are requested with the '''srun''' command.

<pre>
srun --ntasks=1 --cpus-per-task=4 --mem-per-cpu=4096 --time=08:00:00 --partition=medium --job-name=JOB_NAME --pty /bin/bash
</pre>

This command requests for 4 cores (--cpus-per-task) for a single task (--ntasks) with each cpu requesting size 4GB of RAM (--mem-per-cpu) for 8 hrs (--time).

More advanced interactive scenarios to support graphical applications are available using [https://docs.uabgrid.uab.edu/wiki/Setting_Up_VNC_Session VNC] or X11 tunneling [http://www.uab.edu/it/software X-Win32 2014 for Windows]

Interactive jobs that requires running a graphical application, are requested with the '''sinteractive''' command, via '''Terminal''' on your VNC window.
<pre>
sinteractive --ntasks=1 --cpus-per-task=4 --mem-per-cpu=4096 --time=08:00:00 --partition=medium --job-name=JOB_NAME
</pre>
Please note, sinteractive starts your shell in a screen session. Screen is a terminal emulator that is designed to make it possible to detach and reattach a session. This feature can mostly be ignored. If you application uses `ctrl-a` as a special command sequence (e.g. Emacs), however, you may find the application doesn't receive this special character. When using screen, you need to type `ctrl-a a` (ctrl-a followed by a single "a" key press) to send a ctrl-a to your application. Screen uses ctrl-a as it's own command character, so this special sequence issues the command to screen to "send ctrl-a to my app". Learn more about [https://www.gnu.org/software/screen/manual/html_node/Overview.html#Overview screen from it's documentation].

== Storage ==

=== No Automatic Backups ===

There is no automatic back up of any data on the cluster (home, scratch, or whatever). All data back up is managed by you. If you aren't managing a data back up process, then you have no backup data.

=== Home directories ===

Your home directory on Cheaha is NFS-mounted to the compute nodes as /home/$USER or $HOME. It is acceptable to use your home directory as a location to store job scripts, custom code, and libraries. You are responsible for keeping your home directory under 10GB in size!

'''The home directory must not be used to store large amounts of data.''' Please use $USER_SCRATCH
for actively used data sets and $USER_DATA for storage of non scratch data.

=== Scratch ===
Research Computing policy requires that all bulky input and output must be located on the scratch space. The home directory is intended to store your job scripts, log files, libraries and other supporting files.

'''Important Information:'''
* Scratch space (network and local) '''is not backed up'''.
* Research Computing expects each user to keep their scratch areas clean. The cluster scratch area are not to be used for archiving data.

Cheaha has two types of scratch space, network mounted and local.
* Network scratch ($USER_SCRATCH) is available on the login node and each compute node. This storage is a GPFS high performance file system providing roughly 4.7PB of usable storage. This should be your jobs primary working directory, unless the job would benefit from local scratch (see below).
* Local scratch is physically located on each compute node and is not accessible to the other nodes (including the login node). This space is useful if the job performs a lot of file I/O. Most of the jobs that run on our clusters do not fall into this category. Because the local scratch is inaccessible outside the job, it is important to note that you must move any data between local scratch to your network accessible scratch within your job. For example, step 1 in the job could be to copy the input from $USER_SCRATCH to ${USER_SCRATCH}, step 2 code execution, step 3 move the data back to $USER_SCRATCH.

==== Network Scratch ====
Network scratch is available using the environment variable $USER_SCRATCH or directly by /data/scratch/$USER

It is advisable to use the environment variable whenever possible rather than the hard coded path.

==== Local Scratch ====
Each compute node has a local scratch directory that is accessible via the variable '''$LOCAL_SCRATCH'''. If your job performs a lot of file I/O, the job should use $LOCAL_SCRATCH rather than $USER_SCRATCH to prevent bogging down the network scratch file system. The amount of scratch space available is approximately 800GB.

The $LOCAL_SCRATCH is a special temporary directory and it's important to note that this directory is deleted when the job completes, so the job script has to move the results to $USER_SCRATCH or other location prior to the job exiting.

Note that $LOCAL_SCRATCH is only useful for jobs in which all processes run on the same compute node, so MPI jobs are not candidates for this solution.

The following is an array job example that uses $LOCAL_SCRATCH by transferring the inputs into $LOCAL_SCRATCH at the beginning of the script and the result out of $LOCAL_SCRATCH at the end of the script.

<pre>
#!/bin/bash
#SBATCH --array=1-10
#SBATCH --share
#SBATCH --partition=express
#
# Name your job to make it easier for you to track
#
#SBATCH --job-name=R_array_job
#
# Set your error and output files
#
#SBATCH --error=R_array_job.err
#SBATCH --output=R_array_job.out
#SBATCH --ntasks=1
#
# Tell the scheduler only need 10 minutes and the appropriate partition
#
#SBATCH --time=00:10:00
#SBATCH --mem-per-cpu=256
#
# Set your email address and request notification when you job is complete or if it fails
#
#SBATCH --mail-type=FAIL
#SBATCH --mail-user=YOUR_EMAIL_ADDRESS

module load R/3.2.0-goolf-1.7.20

echo "TMPDIR: $LOCAL_SCRATCH"

cd $LOCAL_SCRATCH
# Create a working directory under the special scheduler local scratch directory
# using the array job's taskID
mdkir $SLURM_ARRAY_TASK_ID
cd $SLURM_ARRAY_TASK_ID

# Next copy the input data to the local scratch
echo "Copying input data from network scratch to $LOCAL_SCRATCH/$SLURM_ARRAY_TASK_ID - $(date)
# The input data in this case has a numerical file extension that
# matches $SLURM_ARRAY_TASK_ID
cp -a $USER_SCRATCH/GeneData/INP*.$SLURM_ARRAY_TASK_ID ./
echo "copied input data from network scratch to $LOCAL_SCRATCH/$SLURM_ARRAY_TASK_ID - $(date)

someapp -S 1 -D 10 -i INP*.$SLURM_ARRAY_TASK_ID -o geneapp.out.$SLURM_ARRAY_TASK_ID

# Lastly copy the results back to network scratch
echo "Copying results from local $LOCAL_SCRATCH/$SLURM_ARRAY_TASK_ID to network - $(date)
cp -a geneapp.out.$SLURM_ARRAY_TASK_ID $USER_SCRATCH/GeneData/
echo "Copied results from local $LOCAL_SCRATCH/$SLURM_ARRAY_TASK_ID to network - $(date)

</pre>

=== Project Storage ===
Cheaha has a location where shared data can be stored called $SHARE_PROJECT. As with user scratch, this area '''is not backed up'''!

This is helpful if a team of researchers must access the same data. Please open a help desk ticket to request a project directory under $SHARE_PROJECT.

=== Uploading Data ===

Data can be moved onto the cluster (pushed) from a remote client (ie. you desktop) via SCP or SFTP. Data can also be downloaded to the cluster (pulled) by issuing transfer commands once you are logged into the cluster. Common transfer methods are `wget <URL>`, FTP, or SCP, and depend on how the data is made available from the data provider.

Large data sets should be staged directly to your $USER_SCRATCH directory so as not to fill up $HOME. If you are working on a data set shared with multiple users, it's preferable to request space in $SHARE_PROJECT rather than duplicating the data for each user.

== Environment Modules ==
[http://modules.sourceforge.net/ Environment Modules] is installed on Cheaha and should be used when constructing your job scripts if an applicable module file exists. Using the module command you can easily configure your environment for specific software packages without having to know the specific environment variables and values to set. Modules allows you to dynamically configure your environment without having to logout / login for the changes to take affect.

If you find that specific software does not have a module, please submit a [http://etlab.eng.uab.edu/ helpdesk ticket] to request the module.

* Cheaha supports bash completion for the module command. For example, type 'module' and press the TAB key twice to see a list of options:
<pre>
module TAB TAB

add display initlist keyword refresh switch use
apropos help initprepend list rm unload whatis
avail initadd initrm load show unuse
clear initclear initswitch purge swap update
</pre>

* To see the list of available modulefiles on the cluster, run the '''module avail''' command (note the example list below may not be complete!) or '''module load ''' followed by two tab key presses:
<pre>
module avail

----------------------------------------------------------------------------------------- /cm/shared/modulefiles -----------------------------------------------------------------------------------------
acml/gcc/64/5.3.1 acml/open64-int64/mp/fma4/5.3.1 fftw2/openmpi/gcc/64/float/2.1.5 intel-cluster-runtime/ia32/3.8 netcdf/gcc/64/4.3.3.1
acml/gcc/fma4/5.3.1 blacs/openmpi/gcc/64/1.1patch03 fftw2/openmpi/open64/64/double/2.1.5 intel-cluster-runtime/intel64/3.8 netcdf/open64/64/4.3.3.1
acml/gcc/mp/64/5.3.1 blacs/openmpi/open64/64/1.1patch03 fftw2/openmpi/open64/64/float/2.1.5 intel-cluster-runtime/mic/3.8 netperf/2.7.0
acml/gcc/mp/fma4/5.3.1 blas/gcc/64/3.6.0 fftw3/openmpi/gcc/64/3.3.4 intel-tbb-oss/ia32/44_20160526oss open64/4.5.2.1
acml/gcc-int64/64/5.3.1 blas/open64/64/3.6.0 fftw3/openmpi/open64/64/3.3.4 intel-tbb-oss/intel64/44_20160526oss openblas/dynamic/0.2.15
acml/gcc-int64/fma4/5.3.1 bonnie++/1.97.1 gdb/7.9 iozone/3_434 openmpi/gcc/64/1.10.1
acml/gcc-int64/mp/64/5.3.1 cmgui/7.2 globalarrays/openmpi/gcc/64/5.4 lapack/gcc/64/3.6.0 openmpi/open64/64/1.10.1
acml/gcc-int64/mp/fma4/5.3.1 cuda75/blas/7.5.18 globalarrays/openmpi/open64/64/5.4 lapack/open64/64/3.6.0 pbspro/13.0.2.153173
acml/open64/64/5.3.1 cuda75/fft/7.5.18 hdf5/1.6.10 mpich/ge/gcc/64/3.2 puppet/3.8.4
acml/open64/fma4/5.3.1 cuda75/gdk/352.79 hdf5_18/1.8.16 mpich/ge/open64/64/3.2 rc-base
acml/open64/mp/64/5.3.1 cuda75/nsight/7.5.18 hpl/2.1 mpiexec/0.84_432 scalapack/mvapich2/gcc/64/2.0.2
acml/open64/mp/fma4/5.3.1 cuda75/profiler/7.5.18 hwloc/1.10.1 mvapich/gcc/64/1.2rc1 scalapack/openmpi/gcc/64/2.0.2
acml/open64-int64/64/5.3.1 cuda75/toolkit/7.5.18 intel/compiler/32/15.0/2015.5.223 mvapich/open64/64/1.2rc1 sge/2011.11p1
acml/open64-int64/fma4/5.3.1 default-environment intel/compiler/64/15.0/2015.5.223 mvapich2/gcc/64/2.2b slurm/15.08.6
acml/open64-int64/mp/64/5.3.1 fftw2/openmpi/gcc/64/double/2.1.5 intel-cluster-checker/2.2.2 mvapich2/open64/64/2.2b torque/6.0.0.1

---------------------------------------------------------------------------------------- /share/apps/modulefiles -----------------------------------------------------------------------------------------
rc/BrainSuite/15b rc/freesurfer/freesurfer-5.3.0 rc/intel/compiler/64/ps_2016/2016.0.047 rc/matlab/R2015a rc/SAS/v9.4
rc/cmg/2012.116.G rc/gromacs-intel/5.1.1 rc/Mathematica/10.3 rc/matlab/R2015b
rc/dsistudio/dsistudio-20151020 rc/gtool/0.7.5 rc/matlab/R2012a rc/MRIConvert/2.0.8

--------------------------------------------------------------------------------------- /share/apps/rc/modules/all ---------------------------------------------------------------------------------------
AFNI/linux_openmp_64-goolf-1.7.20-20160616 gperf/3.0.4-intel-2016a MVAPICH2/2.2b-GCC-4.9.3-2.25
Amber/14-intel-2016a-AmberTools-15-patchlevel-13-13 grep/2.15-goolf-1.4.10 NASM/2.11.06-goolf-1.7.20
annovar/2016Feb01-foss-2015b-Perl-5.22.1 GROMACS/5.0.5-intel-2015b-hybrid NASM/2.11.08-foss-2015b
ant/1.9.6-Java-1.7.0_80 GSL/1.16-goolf-1.7.20 NASM/2.11.08-intel-2016a
APBS/1.4-linux-static-x86_64 GSL/1.16-intel-2015b NASM/2.12.02-foss-2016a
ASHS/rev103_20140612 GSL/2.1-foss-2015b NASM/2.12.02-intel-2015b
Aspera-Connect/3.6.1 gtool/0.7.5_linux_x86_64 NASM/2.12.02-intel-2016a
ATLAS/3.10.1-gompi-1.5.12-LAPACK-3.4.2 guile/1.8.8-GNU-4.9.3-2.25 ncurses/5.9-foss-2015b
Autoconf/2.69-foss-2016a HAPGEN2/2.2.0 ncurses/5.9-GCC-4.8.4
Autoconf/2.69-GCC-4.8.4 HarfBuzz/1.2.7-intel-2016a ncurses/5.9-GNU-4.9.3-2.25
Autoconf/2.69-GNU-4.9.3-2.25 HDF5/1.8.15-patch1-intel-2015b ncurses/5.9-goolf-1.4.10
.
.
.
.
</pre>

Some software packages have multiple module files, for example:
* GCC/4.7.2
* GCC/4.8.1
* GCC/4.8.2
* GCC/4.8.4
* GCC/4.9.2
* GCC/4.9.3
* GCC/4.9.3-2.25

In this case, the GCC module will always load the latest version, so loading this module is equivalent to loading GCC/4.9.3-2.25. If you always want to use the latest version, use this approach. If you want use a specific version, use the module file containing the appropriate version number.

Some modules, when loaded, will actually load other modules. For example, the ''GROMACS/5.0.5-intel-2015b-hybrid '' module will also load ''intel/2015b'' and other related tools.

* To load a module, ex: for a GROMACS job, use the following '''module load''' command in your job script:
<pre>
module load GROMACS/5.0.5-intel-2015b-hybrid
</pre>

* To see a list of the modules that you currently have loaded use the '''module list''' command
<pre>
module list

Currently Loaded Modulefiles:
1) slurm/15.08.6 9) impi/5.0.3.048-iccifort-2015.3.187-GNU-4.9.3-2.25 17) Tcl/8.6.3-intel-2015b
2) rc-base 10) iimpi/7.3.5-GNU-4.9.3-2.25 18) SQLite/3.8.8.1-intel-2015b
3) GCC/4.9.3-binutils-2.25 11) imkl/11.2.3.187-iimpi-7.3.5-GNU-4.9.3-2.25 19) Tk/8.6.3-intel-2015b-no-X11
4) binutils/2.25-GCC-4.9.3-binutils-2.25 12) intel/2015b 20) Python/2.7.9-intel-2015b
5) GNU/4.9.3-2.25 13) bzip2/1.0.6-intel-2015b 21) Boost/1.58.0-intel-2015b-Python-2.7.9
6) icc/2015.3.187-GNU-4.9.3-2.25 14) zlib/1.2.8-intel-2015b 22) GROMACS/5.0.5-intel-2015b-hybrid
7) ifort/2015.3.187-GNU-4.9.3-2.25 15) ncurses/5.9-intel-2015b
8) iccifort/2015.3.187-GNU-4.9.3-2.25 16) libreadline/6.3-intel-2015b
</pre>

* A module can be removed from your environment by using the '''module unload''' command:
<pre>
module unload GROMACS/5.0.5-intel-2015b-hybrid
</pre>

* The definition of a module can also be viewed using the '''module show''' command, revealing what a specific module will do to your environment:
<pre>
module show GROMACS/5.0.5-intel-2015b-hybrid
-------------------------------------------------------------------
/share/apps/rc/modules/all/GROMACS/5.0.5-intel-2015b-hybrid:

module-whatis GROMACS is a versatile package to perform molecular dynamics,
i.e. simulate the Newtonian equations of motion for systems with hundreds to millions of particles. - Homepage: http://www.gromacs.org
conflict GROMACS
prepend-path CPATH /share/apps/rc/software/GROMACS/5.0.5-intel-2015b-hybrid/include
prepend-path LD_LIBRARY_PATH /share/apps/rc/software/GROMACS/5.0.5-intel-2015b-hybrid/lib64
prepend-path LIBRARY_PATH /share/apps/rc/software/GROMACS/5.0.5-intel-2015b-hybrid/lib64
prepend-path MANPATH /share/apps/rc/software/GROMACS/5.0.5-intel-2015b-hybrid/share/man
prepend-path PATH /share/apps/rc/software/GROMACS/5.0.5-intel-2015b-hybrid/bin
prepend-path PKG_CONFIG_PATH /share/apps/rc/software/GROMACS/5.0.5-intel-2015b-hybrid/lib64/pkgconfig
setenv EBROOTGROMACS /share/apps/rc/software/GROMACS/5.0.5-intel-2015b-hybrid
setenv EBVERSIONGROMACS 5.0.5
setenv EBDEVELGROMACS /share/apps/rc/software/GROMACS/5.0.5-intel-2015b-hybrid/easybuild/GROMACS-5.0.5-intel-2015b-hybrid-easybuild-devel
-------------------------------------------------------------------
</pre>

=== Error Using Modules from a Job Script ===

If you are using modules and the command your job executes runs fine from the command line but fails when you run it from the job, you may be having an issue with the script initialization. If you see this error in your job error output file
<pre>
-bash: module: line 1: syntax error: unexpected end of file
-bash: error importing function definition for `BASH_FUNC_module'
</pre>
Add the command `unset module` before calling your module files. The -V job argument will cause a conflict with the module function used in your script.

== Sample Job Scripts ==
The following are sample job scripts, please be careful to edit these for your environment (i.e. replace YOUR_EMAIL_ADDRESS with your real email address), set the h_rt to an appropriate runtime limit and modify the job name and any other parameters.

'''Hello World''' is the classic example used throughout programming. We don't want to buck the system, so we'll use it as well to demonstrate jobs submission with one minor variation: our hello world will send us a greeting using the name of whatever machine it runs on. For example, when run on the Cheaha login node, it would print "Hello from login001".

=== Hello World (serial) ===

A serial job is one that can run independently of other commands, ie. it doesn't depend on the data from other jobs running simultaneously. You can run many serial jobs in any order. This is a common solution to processing lots of data when each command works on a single piece of data. For example, running the same conversion on 100s of images.

Here we show how to create job script for one simple command. Running more than one command just requires submitting more jobs.

* Create your hello world application. Run this command to create a script, turn it into to a command, and run the command (just copy and past the following on to the command line).
<pre>
cat > helloworld.sh << EOF
#!/bin/bash
echo Hello from `hostname`
EOF
chmod +x helloworld.sh
./helloworld.sh
</pre>

* Create the Slurm job script that will request 256 MB RAM and a maximum runtime of 10 minutes.
<pre>
$ vi helloworld.job
</pre>
<pre>
#!/bin/bash
#SBATCH --share
#SBATCH --partition=express
#
# Name your job to make it easier for you to track
#
#SBATCH --job-name=helloworld
#
# Set your error and output files
#
#SBATCH --error=helloworld.err
#SBATCH --output=helloworld.out
#SBATCH --ntasks=1
#
# Tell the scheduler only need 10 minutes
#
#SBATCH --time=00:10:00
#SBATCH --mem-per-cpu=256
#
# Set your email address and request notification when you job is complete or if it fails
#
#SBATCH --mail-type=FAIL
#SBATCH --mail-user=$USER@uab.edu

./helloworld.sh
</pre>
* Submit the job to Slurm scheduler and check the status using squeue
<pre>
$ sbatch helloworld.job
Submitted batch job 52888
</pre>
* When the job completes, you should have output files named helloworld.out and helloworld.err
<pre>
$ cat helloworld.out
Hello from c0003
</pre>

=== Hello World (parallel with MPI) ===

MPI is used to coordinate the activity of many computations occurring in parallel. It is commonly used in simulation software for molecular dynamics, fluid dynamics, and similar domains where there is significant communication (data) exchanged between cooperating process.

Here is a simple parallel Slurm job script for running commands the rely on MPI. This example also includes the example of compiling the code and submitting the job script to the Slurm scheduler.

* First, create a directory for the Hello World jobs
<pre>
$ mkdir -p ~/jobs/helloworld
$ cd ~/jobs/helloworld
</pre>
* Create the Hello World code written in C (this example of MPI enabled Hello World includes a 3 minute sleep to ensure the job runs for several minutes, a normal hello world example would run in a matter of seconds).
<pre>
$ vi helloworld-mpi.c
</pre>
<pre>
#include <stdio.h>
#include <mpi.h>

main(int argc, char **argv)
{
int rank, size;

int i, j;
float f;

MPI_Init(&argc,&argv);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);

printf("Hello World from process %d of %d.\n", rank, size);
sleep(180);
for (j=0; j<=100000; j++)
for(i=0; i<=100000; i++)
f=i*2.718281828*i+i+i*3.141592654;

MPI_Finalize();
}
</pre>
* Compile the code, first purging any modules you may have loaded followed by loading the module for OpenMPI GNU. The mpicc command will compile the code and produce a binary named helloworld_gnu_openmpi
<pre>
$ module purge
$ module load OpenMPI/1.8.8-GNU-4.9.3-2.25

$ mpicc helloworld-mpi.c -o helloworld_gnu_openmpi
</pre>
* Create the Slurm job script that will request 8 cpu slots and a maximum runtime of 10 minutes
<pre>
$ vi helloworld.job
</pre>
<pre>
#!/bin/bash
#SBATCH --share
#SBATCH --partition=express
#
# Name your job to make it easier for you to track
#
#SBATCH --job-name=helloworld_mpi
#
# Set your error and output files
#
#SBATCH --error=helloworld_mpi.err
#SBATCH --output=helloworld_mpi.out
#SBATCH --ntasks=8
#
# Tell the scheduler only need 10 minutes
#
#SBATCH --time=00:01:00
#SBATCH --mem-per-cpu=256
#
# Set your email address and request notification when you job is complete or if it fails
#
#SBATCH --mail-type=FAIL
#SBATCH --mail-user=YOUR_EMAIL_ADDRESS

module load OpenMPI/1.8.8-GNU-4.9.3-2.25
mpirun -np $SLURM_NTASKS helloworld_gnu_openmpi
</pre>
* Submit the job to Slurm scheduler and check the status using squeue -u $USER
<pre>
$ sbatch helloworld.job

Submitted batch job 52893

$ squeue -u BLAZERID
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
52893 express hellowor BLAZERID R 2:07 2 c[0005-0006]

</pre>
* When the job completes, you should have output files named helloworld_mpi.out and helloworld_mpi.err
<pre>
$ cat helloworld_mpi.out

Hello World from process 1 of 8.
Hello World from process 3 of 8.
Hello World from process 4 of 8.
Hello World from process 7 of 8.
Hello World from process 5 of 8.
Hello World from process 6 of 8.
Hello World from process 0 of 8.
Hello World from process 2 of 8.
</pre>

=== Hello World (serial) -- revisited ===

The job submit scripts (sbatch scripts) are actually bash shell scripts in their own right. The reason for using the funky #SBATCH prefix in the scripts is so that bash interprets any such line as a comment and won't execute it. Because the # character starts a comment in bash, we can weave the Slurm scheduler directives (the #SBATCH lines) into standard bash scripts. This lets us build scripts that we can execute locally and then easily run the same script to on a cluster node by calling it with sbatch. This can be used to our advantage to create a more fluid experience in moving between development and production job runs.

The following example is a simple variation on the serial job above. All we will do is convert our Slurm job script into a command called helloworld that calls the helloworld.sh command.

If the first line of a file is #!/bin/bash and that file is executable, the shell will automatically run the command as if were any other system command, eg. ls. That is, the ".sh" extension on our HelloWorld.sh script is completely optional and is only meaningful to the user.

Copy the serial helloworld.job script to a new file, add a the special #!/bin/bash as the first line, and make it executable with the following command (note: those are single quotes in the echo command):
<pre>
echo '#!/bin/bash' | cat helloworld.job > helloworld ; chmod +x helloworld
</pre>

Our sbatch script has now become a regular command. We can now execute the command with the simple prefix "./helloworld", which means "execute this file in the current directory":
<pre>
./helloworld
Hello from login001
</pre>
Or if we want to run the command on a compute node, replace the "./" prefix with "sbatch ":
<pre>
$ sbatch helloworld
Submitted batch job 53001
</pre>
And when the cluster run is complete you can look at the content of the output:
<pre>
$ $ cat helloworld.out
Hello from c0003
</pre>

You can use this approach of treating you sbatch files as command wrappers to build a collection of commands that can be executed locally or via sbatch. The other examples can be restructured similarly.

To avoid having to use the "./" prefix, just add the current directory to your PATH. Also, if you plan to do heavy development using this feature on the cluster, please be sure to run [https://docs.uabgrid.uab.edu/wiki/Slurm#Interactive_Session sinteractive] first so you don't load the login node with your development work.

=== Gromacs ===
<pre>
#!/bin/bash
#SBATCH --share
#SBATCH --partition=express
#
# Name your job to make it easier for you to track
#
#SBATCH --job-name=test_gromacs
#
# Set your error and output files
#
#SBATCH --error=test_gromacs.err
#SBATCH --output=test_gromacs.out
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=4
#
# Tell the scheduler only need 10 minutes
#
#SBATCH --time=00:01:00
#SBATCH --mem-per-cpu=256
#
# Set your email address and request notification when you job is complete or if it fails
#
#SBATCH --mail-type=FAIL
#SBATCH --mail-user=YOUR_EMAIL_ADDRESS

module load OpenMPI/1.8.8-GNU-4.9.3-2.25

module load GROMACS/5.0.5-intel-2015b-hybrid

# Change directory to the job working directory if not already there
cd ${USER_SCRATCH}/jobs/gromacs

# Single precision
MDRUN=mdrun_mpi

# Enter your tpr file over here
export MYFILE=example.tpr

srun mpirun $MDRUN -v -s $MYFILE -o $MYFILE -c $MYFILE -x $MYFILE -e $MYFILE -g ${MYFILE}.log

</pre>

=== R ===

The following is an example job script that will use an array of 10 tasks (--array=1-10), each task has a max runtime of 2 hours and will use no more than 256 MB of RAM per task.

Create a working directory and the job submission script
<pre>
$ mkdir -p ~/jobs/ArrayExample
$ cd ~/jobs/ArrayExample
$ vi R-example-array.job
</pre>

<pre>
#!/bin/bash
#SBATCH --array=1-10
#SBATCH --share
#SBATCH --partition=express
#
# Name your job to make it easier for you to track
#
#SBATCH --job-name=R_array_job
#
# Set your error and output files
#
#SBATCH --error=R_array_job.err
#SBATCH --output=R_array_job.out
#SBATCH --ntasks=1
#
# Tell the scheduler only need 10 minutes
#
#SBATCH --time=00:10:00
#SBATCH --mem-per-cpu=256
#
# Set your email address and request notification when you job is complete or if it fails
#
#SBATCH --mail-type=FAIL
#SBATCH --mail-user=YOUR_EMAIL_ADDRESS

module load R/3.2.0-goolf-1.7.20
cd ~/jobs/ArrayExample/rep$SLURM_ARRAY_TASK_ID
srun R CMD BATCH rscript.R
</pre>

Submit the job to the Slurm scheduler and check the status of the job using the squeue command
<pre>
$ sbatch R-example-array.job
$ squeue -u $USER
</pre>

== Installed Software ==

A partial list of installed software with additional instructions for their use is available on the [[Cheaha Software]] page.

Cheaha

2017-10-15T22:04:28Z

Puri@uab.edu: /* Description of Cheaha for Grants */

{{Main_Banner}}
'''Cheaha''' is a campus resource dedicated to enhancing research computing productivity at UAB. [http://cheaha.uabgrid.uab.edu Cheaha] is managed by [http://www.uab.edu/it UAB Information Technology's Research Computing group (UAB ITRC)] and is available to members of the UAB community in need of increased computational capacity. Cheaha supports [http://en.wikipedia.org/wiki/High-performance_computing high-performance computing (HPC)] and [http://en.wikipedia.org/wiki/High-throughput_computing high throughput computing (HTC)] paradigms.

Cheaha provides users with a traditional command-line interactive environment with access to many scientific tools that can leverage its dedicated pool of local compute resources. Alternately, users of graphical applications can start a [[Setting_Up_VNC_Session|cluster desktop]]. The local compute pool provides access to compute hardware based on the [http://en.wikipedia.org/wiki/X86_64 x86-64 64-bit architecture]. The compute resources are organized into a unified Research Computing System. The compute fabric for this system is anchored by the Cheaha cluster, [[ Resources |a commodity cluster with approximately 2400 cores]] connected by low-latency Fourteen Data Rate (FDR) InfiniBand networks. The compute nodes are backed by 6.6PB raw GPFS storage on DDN SFA12KX hardware, an additional 20TB available for home directories on a traditional Hitachi SAN, and other ancillary services. The compute nodes combine to provide over 110TFlops of dedicated computing power.

Cheaha is composed of resources that span data centers located in the UAB Shared Computing facility UAB 936 Building and the RUST Computer Center. Resource design and development is lead by UAB IT Research Computing in open collaboration with community members. Operational [mailto:support@vo.uabgrid.uab.edu support] is provided by UAB IT's Research Computing group.

Cheaha is named in honor of [http://en.wikipedia.org/wiki/Cheaha_Mountain Cheaha Mountain], the highest peak in the state of Alabama. Cheaha is a popular destination whose summit offers clear vistas of the surrounding landscape. (Cheaha Mountain photo-streams on [http://www.flickr.com/search/?q=cheaha Flikr] and [http://picasaweb.google.com/lh/view?q=cheaha&psc=G&filter=1# Picasa]).

== Using ==

=== Getting Started ===

For information on getting an account, logging in, and running a job, please see [[Cheaha2_GettingStarted|Getting Started]].

== History ==

[[Image:Research-computing-platform.png|right|thumb|450px|Logical Diagram of Cheaha Configuration]]

=== 2005 ===

In 2002 UAB was awarded an infrastructure development grant through the NSF EPsCoR program. This led to the 2005 acquisition of a 64 node compute cluster with two AMD Opteron 242 1.6Ghz CPUs per node (128 total cores). This cluster was named Cheaha. Cheaha expanded the compute capacity available at UAB and was the first general-access resource for the community. It lead to expanded roles for UAB IT in research computing support through the development of the UAB Shared HPC Facility in BEC and provided further engagement in Globus-based grid computing resource development on campus via UABgrid and regionally via [http://www.suragrid.org SURAgrid].

=== 2008 ===

In 2008, money was allocated by UAB IT for hardware upgrades which lead to the acquisition of an additional 192 cores based on a Dell clustering solution with Intel Quad-Core E5450 3.0Ghz CPU in August of 2008. This uprade migrated Cheaha's core infrastructure to the Dell blade clustering solution. It provided a 3 fold increase in processor density over the original hardware and enables more computing power to be located in the same physical space with room for expansion, an important consideration in light of the continued growth in processing demand. This hardware represented a major technology upgrade that included space for additional expansion to address over-all capacity demand and enable resource reservation.

The 2008 upgrade began a continuous resource improvement plan that includes a phased development approach for Cheaha with on-going increases in capacity and feature enhancements being brought into production via an [http://projects.uabgrid.uab.edu/cheaha open community process].

Software improvements rolled into the 2008 upgrade included grid computing services to access distributed compute resources and orchestrate jobs using the [http://www.gridway.org GridWay] meta-scheduler. An initial 10Gigabit Ethernet link establishing the UABgrid Research Network was designed to supports high speed data transfers between clusters connected to this network.

=== 2009 ===

In 2009, annual investment funds were directed toward establishing a fully connected dual data rate Infiniband network between the compute nodes added in 2008 and laying the foundation for a research storage system with a 60TB DDN storage system accessed via the Lustre distributed file system. The Infiniband and storage fabrics were designed to support significant increases in research data sets and their associate analytical demand.

=== 2010 ===

In 2010, UAB was awarded an NIH Small Instrumentation Grant (SIG) to further increase analytical and storage capacity. The grant funds were combined with the annual investment funds adding 576 cores (48 nodes) based on the Intel Westmere 2.66 GHz CPU, a quad data rate Infiniband fabric with 32 uplinks, an additional 120 TB of storage for the DDN fabric, and additional hardware to improve reliability. Additional improvements to the research compute platform involved extending the UAB Research Network to link the BEC and RUST data centers, adding 20TB of user and ancillary services storage

=== 2012 ===

In 2012, UAB IT Research Computing invested in the foundation hardware to expand long term storage and virtual machine capabilities with aqcuisition of 12 Dell 720xd system, each containing 16 cores, 96GB RAM, and 36TB of storage, creating a 192 core and 432TB virtual compute and storage fabric.

Additionaly hardware investment by the School of Public Health's Section on Statistical Genetics added three 384GB large memory nodes and an additional 48 cores to the QDR Infiniband fabric.

=== 2013 ===

In 2013, UAB IT Research Computing acquired an [http://blogs.uabgrid.uab.edu/jpr/2013/03/were-going-with-openstack/ OpenStack cloud and Ceph storage software fabric] through a partnership between Dell and Inktank in order to [http://dev.uabgrid.uab.edu extend cloud computing solutions] to the researchers at UAB and enhance the interfacing capabilities for HPC.

=== 2015 ===

UAB IT received $500,000 from the university’s Mission Support Fund for a compute cluster seed expansion of 48 teraflops. This added 936 cores across 40 nodes with 2x12 core 2.5 GHz Intel Xeon E5-2680 v3 compute nodes and FDR InfiniBand interconnect.

UAB received a $500,000 grant from the Alabama Innovation Fund for a three petabyte research storage array. This funding with additional matching from UAB provided a multi-petabyte [https://en.wikipedia.org/wiki/IBM_General_Parallel_File_System GPFS] parallel file system to the cluster which went live in 2016.

=== 2016 ===

In 2016 UAB IT Research computing received additional funding from Deans of CAS, Engineering, and Public Heath to grow the compute capacity provided by the prior year's seed funding. This added an additional compute nodes providing researchers at UAB with a 96 2x12 core (2304 cores total) 2.5 GHz Intel Xeon E5-2680 v3 compute nodes with FDR InfiniBand interconnect. Out of the 96 compute nodes, 36 nodes have 128 GB RAM, 38 nodes have 256 GB RAM, and 14 nodes have 384 GB RAM. There are also four compute nodes with the Intel Xeon Phi 7210 accelerator cards and four compute nodes with the NVIDIA K80 GPUs. More information can be found at [[Resources]].

In addition to the compute, the GPFS six petabyte file system came online. This file system, provided each user five terabyte of personal space, additional space for shared projects and a greatly expanded scratch storage all in a single file system.

The 2015 and 2016 investments combined to provide a completely new core for the Cheaha cluster, allowing the retirement of earlier compute generations.

== Grant and Publication Resources ==

The following description may prove useful in summarizing the services available via Cheaha. If you are using Cheaha for grant funded research please send information about your grant (funding source and grant number), a statement of intent for the research project and a list of the applications you are using to UAB IT Research Computing. If you are using Cheaha for exploratory research, please send a similar note on your research interest. Finally, any publications that rely on computations performed on Cheaha should include a statement acknowledging the use of UAB Research Computing facilities in your research, see the suggested example below. Please note, your acknowledgment may also need to include an addition statement acknowledging grant-funded hardware. We also ask that you send any references to publications based on your use of Cheaha compute resources.

=== Description of Cheaha for Grants ===

UAB IT Research Computing maintains high performance compute and storage resources for investigators. The Cheaha compute cluster provides 2800 conventional CPU cores and 80 accelerators interconnected via InfiniBand network and provides 468 TFLOP/s of aggregate theoretical peak performance. A high-performance, 6.6PB raw GPFS storage on DDN SFA12KX hardware is also connected to these compute nodes via the Infiniband fabric. An additional 20TB of traditional SAN storage is also available for home directories. This general access compute fabric is available to all UAB investigators.

=== Acknowledgment in Publications ===

This work was supported in part by the National Science Foundation under Grants Nos. OAC-1541310, the University of Alabama at Birmingham, and the Alabama Innovation Fund. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation or the University of Alabama at Birmingham.

== System Profile ==

=== Performance ===
{{CheahaTflops}}

=== Hardware ===

The Cheaha Compute Platform includes three generations of commodity compute hardware, totaling 868 compute cores, 2.8TB of RAM, and over 200TB of storage.

The hardware is grouped into generations designated gen1, gen2, and gen3 (oldest to newest). The following descriptions highlight the hardware profile for each generation.

* Generation 1 (gen1) -- 64 2-CPU AMD 1.6 GHz compute nodes with Gigabit interconnect. This is the original hardware collection purchased with NSF EPSCoR funds in 2005, approx $150K. These nodes are sometimes called the "Verari" nodes. These nodes are tagged as "verari-compute-#-#" in the ROCKS naming convention.
* Generation 2 (gen2) -- 24 2x4 core (196 cores total) Intel 3.0 GHz Intel compute nodes with dual data rate Infiniband interconnect and the initial high-perf storage implementation using 60TB DDN. This is the hardware collection purchased exclusively with the annual VPIT funds allocation, approx $150K/yr for the 2008 and 2009 fiscal years. These nodes are sometimes confusingly called "cheaha2" or "cheaha" nodes. These nodes are tagged as "cheaha-compute-#-#" in the ROCKS naming convention.
* Generation 3 (gen3) -- 48 2x6 core (576 cores total) 2.66 GHz Intel compute nodes with quad data rate Infiniband, ScaleMP, and the high-perf storage build-out for capacity and redundancy with 120TB DDN. This is the hardware collection purchased with a combination of the NIH SIG funds and some of the 2010 annual VPIT investment. These nodes were given the code name "sipsey" and tagged as such in the node naming for the queue system. These nodes are tagged as "sipsey-compute-#-#" in the ROCKS naming convention. 16 of the gen3 nodes (sipsey-compute-0-1 thru sipsey-compute-0-16) were upgraded in 2014 from 48GB to 96GB of memory per node.
* Generation 4 (gen4) -- 3 16 core (48 cores total) compute nodes. This hardware collection purchase by [http://www.soph.uab.edu/ssg/people/tiwari Dr. Hemant Tiwari of SSG]. These nodes were given the code name "ssg" and tagged as such in the node naming for the queue system. These nodes are tagged as "ssg-compute-0-#" in the ROCKS naming convention.
* Generation 6 (gen6) --
** 44 Compute Nodes with two 12 core processors (Intel Xeon E5-2680 v3 2.5GHz) with 128GB DDR4 RAM, FDR InfiniBand and 10GigE network cards (4 nodes with NVIDIA K80 GPUs and 4 nodes with Intel Xeon Phi 7120P accelerators)
** 38 Compute Nodes with two 12 core processors (Intel Xeon E5-2680 v3 2.5GHz) with 256GB DDR4 RAM, FDR InfiniBand and 10GigE network cards
** 14 Compute Nodes with two 12 core processors (Intel Xeon E5-2680 v3 2.5GHz) with 384GB DDR4 RAM, FDR InfiniBand and 10GigE network card
** FDR InfiniBand Switch
** 10Gigabit Ethernet Switch
** Management node and gigabit switch for cluster management
** Bright Advanced Cluster Management software licenses

Summarized, Cheaha's compute pool includes:
* gen4 is 48 cores of [http://ark.intel.com/products/64583/Intel-Xeon-Processor-E5-2680-20M-Cache-2_70-GHz-8_00-GTs-Intel-QPI 2.70GHz eight-core Intel Xeon E5-2680 processors] with 24G of RAM per core or 384GB total
* gen3 is 192 cores of [http://ark.intel.com/products/47922/Intel-Xeon-Processor-X5650-12M-Cache-2_66-GHz-6_40-GTs-Intel-QPI?q=x5650 2.67GHz six-core Intel Xeon X5650 processors] with 8Gb RAM per core or 96GB total
* gen3 is 384 cores of [http://ark.intel.com/products/47922/Intel-Xeon-Processor-X5650-12M-Cache-2_66-GHz-6_40-GTs-Intel-QPI?q=x5650 2.67GHz six-core Intel Xeon X5650 processors] with 4Gb RAM per core or 48GB total
* gen2 is 192 cores of [http://ark.intel.com/products/33083/Intel-Xeon-Processor-E5450-12M-Cache-3_00-GHz-1333-MHz-FSB 3.0GHz quad-core Intel Xeon E5450 processors] with 2Gb RAM per core
* gen1 is 100 cores of 1.6GhZ AMD Opteron 242 processors with 1Gb RAM per core

{|border="1" cellpadding="2" cellspacing="0"
|+ Physical Nodes
|- bgcolor=grey
!gen!!queue!!#nodes!!cores/node!!RAM/node
|-
|gen6|| default || 44 || 24 || 128G
|-
|gen6|| default || 38 || 24 || 256G
|-
|gen6|| default || 14 || 24 || 384G
|-
|gen5||Ceph/OpenStack|| 12 || 20 || 96G
|-
|gen4||ssg||3||16||385G
|-
|gen3||sipsey||16||12||96G
|-
|gen3||sipsey||32||12||48G
|-
|gen2||cheaha||24||8||16G
|}

=== Software ===

Details of the software available on Cheaha can be found on the [http://me.eng.uab.edu/wiki/index.php?title=Cheaha#Installed_software Cheaha cluster configuration page], an overview follows.

Cheaha uses [http://modules.sourceforge.net/ Environment Modules] to support account configuration. Please follow these [http://me.eng.uab.edu/wiki/index.php?title=Cheaha#Environment_Modules specific steps for using environment modules].

Cheaha's software stack is built with the [http://www.brightcomputing.com Bright Cluster Manager]. Cheaha's operating system is CentOS with the following major cluster components:
* BrightCM 7.2
* CentOS 7.2 x86_64
* [[Slurm]] 15.08

A brief summary of the some of the available computational software and tools available includes:
* Amber
* FFTW
* Gromacs
* GSL
* NAMD
* VMD
* Intel Compilers
* GNU Compilers
* Java
* R
* OpenMPI
* MATLAB

=== Network ===

Cheaha is connected to the UAB Research Network which provides a dedicated 10Gbs networking backplane between clusters located in the 936 data center and the campus network core. Data transfers rates of almost 8Gbps between these hosts have been demonstrated using Grid FTP, a multi-channel file transfer service that is used to move data between clusters as part of the job management operations. This performance promises very efficient job management and the seamless integration of other clusters as connectivity to the research network is expanded.

=== Benchmarks ===

The continuous resource improvement process involves collecting benchmarks of the system. One of the measures of greatest interest to users of the system are benchmarks of specific application codes. The following benchmarks have been performed on the system and will be further expanded as additional benchmarks are performed.

* [[Cheaha-BGL_Comparison|Cheaha-BGL Comparison]]

* [[Gromacs_Benchmark|Gromacs]]

* [[NAMD_Benchmarks|NAMD]]

=== Cluster Usage Statistics ===

Cheaha uses Bright Cluster Manager to report cluster performance data. This information provides a helpful overview of the current and historical operating stats for Cheaha. You can access the status monitoring page [https://cheaha-master01.rc.uab.edu/userportal/ here] (accessible only on the UAB network or through VPN).

== Availability ==

Cheaha is a general-purpose computer resource made available to the UAB community by UAB IT. As such, it is available for legitimate research and educational needs and is governed by [http://www.uabgrid.uab.edu/aup UAB's Acceptable Use Policy (AUP)] for computer resources.

Many software packages commonly used across UAB are available via Cheaha.

To request access to Cheaha, please send a request to [mailto:support@vo.uabgrid.uab.edu send a request] to the cluster support group.

Cheaha's intended use implies broad access to the community, however, no guarantees are made that specific computational resources will be available to all users. Availability guarantees can only be made for reserved resources.

=== Secure Shell Access ===

Please configure you client secure shell software to use the official host name to access Cheaha:

<pre>
cheaha.rc.uab.edu
</pre>

== Scheduling Framework ==

[http://slurm.schedmd.com/ Slurm] is a queue management system and stands for Simple Linux Utility for Resource Management. Slurm was developed at the Lawrence Livermore National Lab and currently runs some of the largest compute clusters in the world. '''[[Slurm]]''' is now the primary job manager on Cheaha, it replaces SUN Grid Engine (SGE) the job manager used earlier.

Slurm is similar in many ways to GridEngine or most other queue systems. You write a batch script then submit it to the queue manager (scheduler). The queue manager then schedules your job to run on the queue (or '''partition''' in Slurm parlance) that you designate. Below we will provide an outline of how to submit jobs to Slurm, how Slurm decides when to schedule your job, and how to monitor progress.

== Support ==

Operational support for Cheaha is provided by the Research Computing group in UAB IT. For questions regarding the operational status of Cheaha please send your request to [mailto:support@vo.uabgrid.uab.edu support@vo.uabgrid.uab.edu]. As a user of Cheaha you will automatically by subscribed to the hpc-announce email list. This subscription is mandatory for all users of Cheaha. It is our way of communicating important information regarding Cheaha to you. The traffic on this list is restricted to official communication and has a very low volume.

We have limited capacity, however, to support non-operational issue like "How do I write a job script" or "How do I compile a program". For such requests, you may find it more fruitful to send your questions to the hpc-users email list and request help from our peers in the HPC community at UAB. As with all mailing lists, please observe [http://lifehacker.com/5473859/basic-etiquette-for-email-lists-and-forums common mailing etiquette].

Finally, please remember that as you learned about HPC from others it becomes part of your responsibilty to help others on their quest. You should update this documentation or respond to mailing list requests of others.

You can subscribe to hpc-users by sending an email to:

[mailto:sympa@vo.uabgrid.uab.edu?subject=subscribe%20hpc-users sympa@vo.uabgrid.uab.edu with the subject ''subscribe hpc-users''].

You can unsubribe from hpc-users by sending an email to:

[mailto:sympa@vo.uabgrid.uab.edu?subject=unsubscribe%20hpc-users sympa@vo.uabgrid.uab.edu with the subject ''unsubscribe hpc-users''].

You can review archives of the list in the [http://vo.uabgrid.uab.edu/sympa/arc/hpc-users web hpc-archives].

If you need help using the list service please send an email to:

[mailto:sympa@vo.uabgrid.uab.edu?subject=help sympa@vo.uabgrid.uab.edu with the subject ''help'']

If you have questions about the operation of the list itself, please send an email to the owners of the list:

[mailto:hpc-users-request@vo.uabgrid.uab.edu sympa@vo.uabgrid.uab.edu with a subject relavent to your issue with the list]

If you are interested in contributing to the enhancement of HPC features at UAB or would like to talk to other cluster administrators, [mailto:sympa@vo.uabgrid.uab.edu?subject=subscribe%20hpc-dev please join the hpc developers community at UAB].

Template:CheahaTflops

2017-10-15T22:02:16Z

Puri@uab.edu:

{| class="wikitable"
|-
!Generation!! Type!! Nodes!! CPUs per Node!! Cores Per CPU!! Total Cores!!Clock Speed (GHz)!! Instructions Per Cycle !!Hardware Reference
|-
|Gen 2|| Intel Xeon E5450||align="right"| 24||align="right"| 2||align="right"| 4||align="right"| 192||align="right"| 3.00||align="right"| 4||[http://ark.intel.com/products/33083/Intel-Xeon-Processor-E5450-(12M-Cache-3_00-GHz-1333-MHz-FSB) Intel Xeon E5450]
|-
|Gen 3|| Intel Xeon X5650||align="right"| 48||align="right"| 2||align="right"| 6||align="right"| 576||align="right"| 2.66||align="right"| 4|| [http://ark.intel.com/products/47922/Intel-Xeon-Processor-X5650-(12M-Cache-2_66-GHz-6_40-GTs-Intel-QPI) Intel Xeon E6550]
|-
|Gen 4|| Intel Xeon E5-2680||align="right"| 3||align="right"| 2||align="right"| 8||align="right"| 48||align="right"| 2.70||align="right"| 8|| [http://ark.intel.com/products/64583 Intel Xeon E2680]
|-
|Gen 5|| Intel Xeon E5-2650||align="right"| 12||align="right"| 2||align="right"| 8||align="right"| 192||align="right"| 2.00||align="right"| 8|| [http://ark.intel.com/products/64590/Intel-Xeon-Processor-E5-2650-20M-Cache-2_00-GHz-8_00-GTs-Intel-QPI Intel Xeon E2650] (Nodes dedicated to [https://dev.uabgrid.uab.edu/wiki/OpenStackPlusCeph OpenStack+Ceph] with 10Gbs network)
|-
|Gen 6†|| Intel Xeon E5-2680 v3||align="right"| 96||align="right"| 2||align="right"| 12||align="right"| 2304||align="right"| 2.50||align="right"| 16|| [http://ark.intel.com/products/81908/Intel-Xeon-Processor-E5-2680-v3-30M-Cache-2_50-GHz Intel Xeon E5-2680 v3]
|-
|Gen 7††|| Intel Xeon E5-2680 v4||align="right"| 18||align="right"| 2||align="right"| 14||align="right"| 504||align="right"| 2.40||align="right"| 16|| [https://ark.intel.com/products/91754/Intel-Xeon-Processor-E5-2680-v4-35M-Cache-2_40-GHz Intel Xeon E5-2680 v4]

|}

{| class="wikitable"
|+[http://en.wikipedia.org/wiki/FLOPS#Computing Theoretical Peak Flops] = (number of cores) * (clock speed) * (instructions per cycle)
|-
!Generation!!Theoretical Peak Tera-FLOPS
|-
|Gen 2||align="right"| 2.304
|-
|Gen 3||align="right"| 6.129
|-
|Gen 4||align="right"| 1.036
|-
|Gen 5||align="right"| 3.072
|-
|Gen 6†||align="right"| 110
|-
|Gen 7††||align="right"| 358
|-
|}

† Includes four Intel Xeon Phi 7210 accelerators and four NVIDIA K80 GPUs. 
†† Includes 72 [https://images.nvidia.com/content/tesla/pdf/nvidia-tesla-p100-PCIe-datasheet.pdf NVIDIA Tesla P100 16GB] GPUs.

Puri@uab.edu: /* Software */

{{Main_Banner}}
'''Cheaha''' is a campus resource dedicated to enhancing research computing productivity at UAB. [http://cheaha.uabgrid.uab.edu Cheaha] is managed by [http://www.uab.edu/it UAB Information Technology's Research Computing group (UAB ITRC)] and is available to members of the UAB community in need of increased computational capacity. Cheaha supports [http://en.wikipedia.org/wiki/High-performance_computing high-performance computing (HPC)] and [http://en.wikipedia.org/wiki/High-throughput_computing high throughput computing (HTC)] paradigms.

Cheaha provides users with a traditional command-line interactive environment with access to many scientific tools that can leverage its dedicated pool of local compute resources. Alternately, users of graphical applications can start a [[Setting_Up_VNC_Session|cluster desktop]]. The local compute pool provides access to three generations of compute hardware based on the [http://en.wikipedia.org/wiki/X86_64 x86-64 64-bit architecture]. The compute resources are organized into a unified Research Computing System. The compute fabric for this system is anchored by the Cheaha cluster, [[ Resources |a commodity cluster with several generations of hardware with approximately 3000 cores]] connected by low-latency Fourteen Data Rate (FDR) and Quad Data Rate (QDR) InfiniBand networks. The compute nodes are backed by 6.6PB raw GPFS storage on DDN SFA12KX hardware, a 180TB of high performance Lustre storage on a DDN hardware and an additional 20TB available for home directories on a traditional Hitachi SAN and other ancillary services. The compute nodes combine to provide over 120TFlops of dedicated computing power.

Cheaha is composed of resources that span data centers located in the UAB Shared Computing facility UAB 936 Building and the RUST Computer Center. Resource design and development is lead by UAB IT Research Computing in open collaboration with community members. Operational [mailto:support@vo.uabgrid.uab.edu support] is provided by UAB IT's Research Computing group.

Cheaha is named in honor of [http://en.wikipedia.org/wiki/Cheaha_Mountain Cheaha Mountain], the highest peak in the state of Alabama. Cheaha is a popular destination whose summit offers clear vistas of the surrounding landscape. (Cheaha Mountain photo-streams on [http://www.flickr.com/search/?q=cheaha Flikr] and [http://picasaweb.google.com/lh/view?q=cheaha&psc=G&filter=1# Picasa]).

== Using ==

=== Getting Started ===

For information on getting an account, logging in, and running a job, please see [[Cheaha2_GettingStarted|Getting Started]].

== History ==

[[Image:Research-computing-platform.png|right|thumb|450px|Logical Diagram of Cheaha Configuration]]

=== 2005 ===

In 2002 UAB was awarded an infrastructure development grant through the NSF EPsCoR program. This led to the 2005 acquisition of a 64 node compute cluster with two AMD Opteron 242 1.6Ghz CPUs per node (128 total cores). This cluster was named Cheaha. Cheaha expanded the compute capacity available at UAB and was the first general-access resource for the community. It lead to expanded roles for UAB IT in research computing support through the development of the UAB Shared HPC Facility in BEC and provided further engagement in Globus-based grid computing resource development on campus via UABgrid and regionally via [http://www.suragrid.org SURAgrid].

=== 2008 ===

In 2008, money was allocated by UAB IT for hardware upgrades which lead to the acquisition of an additional 192 cores based on a Dell clustering solution with Intel Quad-Core E5450 3.0Ghz CPU in August of 2008. This uprade migrated Cheaha's core infrastructure to the Dell blade clustering solution. It provided a 3 fold increase in processor density over the original hardware and enables more computing power to be located in the same physical space with room for expansion, an important consideration in light of the continued growth in processing demand. This hardware represented a major technology upgrade that included space for additional expansion to address over-all capacity demand and enable resource reservation.

The 2008 upgrade began a continuous resource improvement plan that includes a phased development approach for Cheaha with on-going increases in capacity and feature enhancements being brought into production via an [http://projects.uabgrid.uab.edu/cheaha open community process].

Software improvements rolled into the 2008 upgrade included grid computing services to access distributed compute resources and orchestrate jobs using the [http://www.gridway.org GridWay] meta-scheduler. An initial 10Gigabit Ethernet link establishing the UABgrid Research Network was designed to supports high speed data transfers between clusters connected to this network.

=== 2009 ===

In 2009, annual investment funds were directed toward establishing a fully connected dual data rate Infiniband network between the compute nodes added in 2008 and laying the foundation for a research storage system with a 60TB DDN storage system accessed via the Lustre distributed file system. The Infiniband and storage fabrics were designed to support significant increases in research data sets and their associate analytical demand.

=== 2010 ===

In 2010, UAB was awarded an NIH Small Instrumentation Grant (SIG) to further increase analytical and storage capacity. The grant funds were combined with the annual investment funds adding 576 cores (48 nodes) based on the Intel Westmere 2.66 GHz CPU, a quad data rate Infiniband fabric with 32 uplinks, an additional 120 TB of storage for the DDN fabric, and additional hardware to improve reliability. Additional improvements to the research compute platform involved extending the UAB Research Network to link the BEC and RUST data centers, adding 20TB of user and ancillary services storage

=== 2012 ===

In 2012, UAB IT Research Computing invested in the foundation hardware to expand long term storage and virtual machine capabilities with aqcuisition of 12 Dell 720xd system, each containing 16 cores, 96GB RAM, and 36TB of storage, creating a 192 core and 432TB virtual compute and storage fabric.

Additionaly hardware investment by the School of Public Health's Section on Statistical Genetics added three 384GB large memory nodes and an additional 48 cores to the QDR Infiniband fabric.

=== 2013 ===

In 2013, UAB IT Research Computing acquired an [http://blogs.uabgrid.uab.edu/jpr/2013/03/were-going-with-openstack/ OpenStack cloud and Ceph storage software fabric] through a partnership between Dell and Inktank in order to extend cloud computing solutions to the researchers at UAB and enhance the interfacing capabilities for HPC. This fabric is under [http://dev.uabgrid.uab.edu active development] and will see feature releases in the coming months.

=== 2015/2016 ===

In 2015/2016 UAB IT Research computing acquired 96 2x12 core (2304 cores total) 2.5 GHz Intel Xeon E5-2680 v3 compute nodes with FDR InfiniBand interconnect. Out of the 96 compute nodes, 36 nodes have 128 GB RAM, 38 nodes have 256 GB RAM, and 14 nodes have 384 GB RAM. There are also four compute nodes with the Intel Xeon Phi 7210 accelerator cards and four compute nodes with the NVIDIA K80 GPUs. More information can be found at [[Resources]].

== Grant and Publication Resources ==

The following description may prove useful in summarizing the services available via Cheaha. If you are using Cheaha for grant funded research please send information about your grant (funding source and grant number), a statement of intent for the research project and a list of the applications you are using to UAB IT Research Computing. If you are using Cheaha for exploratory research, please send a similar note on your research interest. Finally, any publications that rely on computations performed on Cheaha should include a statement acknowledging the use of UAB Research Computing facilities in your research, see the suggested example below. Please note, your acknowledgment may also need to include an addition statement acknowledging grant-funded hardware. We also ask that you send any references to publications based on your use of Cheaha compute resources.

=== Description of Cheaha for Grants ===

UAB IT Research Computing maintains high performance compute and storage resources for investigators. The Cheaha compute cluster provides 3120 conventional CPU cores across five generations of hardware that provide over 120 TFLOP/s of combined computational performance, and 20 TB of system memory interconnected via an Infiniband network. A high-performance, 6.6PB raw GPFS storage on DDN SFA12KX hardware and 180TB Lustre parallel file system built on a Direct Data Network (DDN) hardware platform is also connected to these cores via the Infiniband fabric. An additional 20TB of traditional SAN storage and 432TB of OpenStack+Ceph storage is available via a 10+ GigE network fabric. This general access compute fabric is available to all UAB investigators.

Additionally, NIH funded investigators are granted priority access to the NIH SIG award acquired compute and storage pool that includes an additional 576 2.66GHz Intel-based compute cores, 2.3TB RAM and 180TB in the high-performance Lustre parallel file system all interconnected via a QDR Infiniband network fabric.

=== Acknowledgment in Publications ===

This work was supported in part by the research computing resources acquired and managed by UAB IT Research Computing. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the University of Alabama at Birmingham.

Additionally, NIH funded researchers using the cluster for genomics or proteomics analyses should include the following statement regarding the NIH funded gen3 (SIG) compute resources.

"Computational portions of this research were supported by NIH S10RR026723. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Institutes of Health"

== System Profile ==

=== Performance ===
{{CheahaTflops}}

=== Hardware ===

The Cheaha Compute Platform includes three generations of commodity compute hardware, totaling 868 compute cores, 2.8TB of RAM, and over 200TB of storage.

The hardware is grouped into generations designated gen1, gen2, and gen3 (oldest to newest). The following descriptions highlight the hardware profile for each generation.

* <strike> Generation 1 (gen1) -- 64 2-CPU AMD 1.6 GHz compute nodes with Gigabit interconnect. This is the original hardware collection purchased with NSF EPSCoR funds in 2005, approx $150K. These nodes are sometimes called the "Verari" nodes. These nodes are tagged as "verari-compute-#-#" in the ROCKS naming convention.</strike> (gen1 was decommissioned in June 2013)
* Generation 2 (gen2) -- 24 2x4 core (196 cores total) Intel 3.0 GHz Intel compute nodes with dual data rate Infiniband interconnect and the initial high-perf storage implementation using 60TB DDN. This is the hardware collection purchased exclusively with the annual VPIT funds allocation, approx $150K/yr for the 2008 and 2009 fiscal years. These nodes are sometimes confusingly called "cheaha2" or "cheaha" nodes. These nodes are tagged as "cheaha-compute-#-#" in the ROCKS naming convention.
* Generation 3 (gen3) -- 48 2x6 core (576 cores total) 2.66 GHz Intel compute nodes with quad data rate Infiniband, ScaleMP, and the high-perf storage build-out for capacity and redundancy with 120TB DDN. This is the hardware collection purchased with a combination of the NIH SIG funds and some of the 2010 annual VPIT investment. These nodes were given the code name "sipsey" and tagged as such in the node naming for the queue system. These nodes are tagged as "sipsey-compute-#-#" in the ROCKS naming convention. 16 of the gen3 nodes (sipsey-compute-0-1 thru sipsey-compute-0-16) were upgraded in 2014 from 48GB to 96GB of memory per node.
* Generation 4 (gen4) -- 3 16 core (48 cores total) compute nodes. This hardware collection purchase by [http://www.soph.uab.edu/ssg/people/tiwari Dr. Hemant Tiwari of SSG]. These nodes were given the code name "ssg" and tagged as such in the node naming for the queue system. These nodes are tagged as "ssg-compute-0-#" in the ROCKS naming convention.
* Generation 6 (gen6) --
** 44 Compute Nodes with two 12 core processors (Intel Xeon E5-2680 v3 2.5GHz) with 128GB DDR4 RAM, FDR InfiniBand and 10GigE network cards (4 nodes with NVIDIA K80 GPUs and 4 nodes with Intel Xeon Phi 7120P accelerators)
** 38 Compute Nodes with two 12 core processors (Intel Xeon E5-2680 v3 2.5GHz) with 256GB DDR4 RAM, FDR InfiniBand and 10GigE network cards
** 14 Compute Nodes with two 12 core processors (Intel Xeon E5-2680 v3 2.5GHz) with 384GB DDR4 RAM, FDR InfiniBand and 10GigE network card
** FDR InfiniBand Switch
** 10Gigabit Ethernet Switch
** Management node and gigabit switch for cluster management
** Bright Advanced Cluster Management software licenses

Summarized, Cheaha's compute pool includes:
* gen4 is 48 cores of [http://ark.intel.com/products/64583/Intel-Xeon-Processor-E5-2680-20M-Cache-2_70-GHz-8_00-GTs-Intel-QPI 2.70GHz eight-core Intel Xeon E5-2680 processors] with 24G of RAM per core or 384GB total
* gen3 is 192 cores of [http://ark.intel.com/products/47922/Intel-Xeon-Processor-X5650-12M-Cache-2_66-GHz-6_40-GTs-Intel-QPI?q=x5650 2.67GHz six-core Intel Xeon X5650 processors] with 8Gb RAM per core or 96GB total
* gen3 is 384 cores of [http://ark.intel.com/products/47922/Intel-Xeon-Processor-X5650-12M-Cache-2_66-GHz-6_40-GTs-Intel-QPI?q=x5650 2.67GHz six-core Intel Xeon X5650 processors] with 4Gb RAM per core or 48GB total
* gen2 is 192 cores of [http://ark.intel.com/products/33083/Intel-Xeon-Processor-E5450-12M-Cache-3_00-GHz-1333-MHz-FSB 3.0GHz quad-core Intel Xeon E5450 processors] with 2Gb RAM per core
* <strike> gen1 is 100 cores of 1.6GhZ AMD Opteron 242 processors with 1Gb RAM per core </strike> (decomissioned June 2013)

{|border="1" cellpadding="2" cellspacing="0"
|+ Physical Nodes
|- bgcolor=grey
!gen!!queue!!#nodes!!cores/node!!RAM/node
|-
|gen6|| default || 44 || 24 || 128G
|-
|gen6|| default || 38 || 24 || 256G
|-
|gen6|| default || 14 || 24 || 384G
|-
|gen5||Ceph/OpenStack|| 12 || 20 || 96G
|-
|gen4||ssg||3||16||385G
|-
|gen3||sipsey||16||12||96G
|-
|gen3||sipsey||32||12||48G
|-
|gen2||cheaha||24||8||16G
|}

=== Software ===

Details of the software available on Cheaha can be found on the [http://me.eng.uab.edu/wiki/index.php?title=Cheaha#Installed_software Cheaha cluster configuration page], an overview follows.

Cheaha uses [http://modules.sourceforge.net/ Environment Modules] to support account configuration. Please follow these [http://me.eng.uab.edu/wiki/index.php?title=Cheaha#Environment_Modules specific steps for using environment modules].

Cheaha's software stack is built with the [http://www.brightcomputing.com Bright Cluster Manager]. Cheaha's operating system is CentOS with the following major cluster components:
* BrightCM 7.2
* CentOS 7.2 x86_64
* [[Slurm]] 15.08

A brief summary of the some of the available computational software and tools available includes:
* Amber
* FFTW
* Gromacs
* GSL
* NAMD
* VMD
* Intel Compilers
* GNU Compilers
* Java
* R
* OpenMPI
* MATLAB

=== Network ===

Cheaha is connected to the UAB Research Network which provides a dedicated 10Gbs networking backplane between clusters located in the 936 data center and the campus network core. Data transfers rates of almost 8Gbps between these hosts have been demonstrated using Grid FTP, a multi-channel file transfer service that is used to move data between clusters as part of the job management operations. This performance promises very efficient job management and the seamless integration of other clusters as connectivity to the research network is expanded.

=== Benchmarks ===

The continuous resource improvement process involves collecting benchmarks of the system. One of the measures of greatest interest to users of the system are benchmarks of specific application codes. The following benchmarks have been performed on the system and will be further expanded as additional benchmarks are performed.

* [[Cheaha-BGL_Comparison|Cheaha-BGL Comparison]]

* [[Gromacs_Benchmark|Gromacs]]

* [[NAMD_Benchmarks|NAMD]]

=== Performance Statistics ===

Cheaha uses Ganglia to report cluster performance data. This information provides a helpful overview of the current and historical operating stats for Cheaha. You can access the Ganglia monitoring page [https://cheaha.uabgrid.uab.edu/ganglia/ here].

== Availability ==

Cheaha is a general-purpose computer resource made available to the UAB community by UAB IT. As such, it is available for legitimate research and educational needs and is governed by [http://www.uabgrid.uab.edu/aup UAB's Acceptable Use Policy (AUP)] for computer resources.

Many software packages commonly used across UAB are available via Cheaha.

To request access to Cheaha, please send a request to [mailto:support@vo.uabgrid.uab.edu send a request] to the cluster support group.

Cheaha's intended use implies broad access to the community, however, no guarantees are made that specific computational resources will be available to all users. Availability guarantees can only be made for reserved resources.

=== Secure Shell Access ===

Please configure you client secure shell software to use the official host name to access Cheaha:

<pre>
cheaha.rc.uab.edu
</pre>

== Scheduling Framework ==

[http://slurm.schedmd.com/ Slurm] is a queue management system and stands for Simple Linux Utility for Resource Management. Slurm was developed at the Lawrence Livermore National Lab and currently runs some of the largest compute clusters in the world. '''[[Slurm]]''' is now the primary job manager on Cheaha, it replaces SUN Grid Engine (SGE) the job manager used earlier.

Slurm is similar in many ways to GridEngine or most other queue systems. You write a batch script then submit it to the queue manager (scheduler). The queue manager then schedules your job to run on the queue (or '''partition''' in Slurm parlance) that you designate. Below we will provide an outline of how to submit jobs to Slurm, how Slurm decides when to schedule your job, and how to monitor progress.

== Support ==

Operational support for Cheaha is provided by the Research Computing group in UAB IT. For questions regarding the operational status of Cheaha please send your request to [mailto:support@vo.uabgrid.uab.edu support@vo.uabgrid.uab.edu]. As a user of Cheaha you will automatically by subscribed to the hpc-announce email list. This subscription is mandatory for all users of Cheaha. It is our way of communicating important information regarding Cheaha to you. The traffic on this list is restricted to official communication and has a very low volume.

We have limited capacity, however, to support non-operational issue like "How do I write a job script" or "How do I compile a program". For such requests, you may find it more fruitful to send your questions to the hpc-users email list and request help from our peers in the HPC community at UAB. As with all mailing lists, please observe [http://lifehacker.com/5473859/basic-etiquette-for-email-lists-and-forums common mailing etiquette].

Finally, please remember that as you learned about HPC from others it becomes part of your responsibilty to help others on their quest. You should update this documentation or respond to mailing list requests of others.

You can subscribe to hpc-users by sending an email to:

[mailto:sympa@vo.uabgrid.uab.edu?subject=subscribe%20hpc-users sympa@vo.uabgrid.uab.edu with the subject ''subscribe hpc-users''].

You can unsubribe from hpc-users by sending an email to:

[mailto:sympa@vo.uabgrid.uab.edu?subject=unsubscribe%20hpc-users sympa@vo.uabgrid.uab.edu with the subject ''unsubscribe hpc-users''].

You can review archives of the list in the [http://vo.uabgrid.uab.edu/sympa/arc/hpc-users web hpc-archives].

If you need help using the list service please send an email to:

[mailto:sympa@vo.uabgrid.uab.edu?subject=help sympa@vo.uabgrid.uab.edu with the subject ''help'']

If you have questions about the operation of the list itself, please send an email to the owners of the list:

[mailto:hpc-users-request@vo.uabgrid.uab.edu sympa@vo.uabgrid.uab.edu with a subject relavent to your issue with the list]

If you are interested in contributing to the enhancement of HPC features at UAB or would like to talk to other cluster administrators, [mailto:sympa@vo.uabgrid.uab.edu?subject=subscribe%20hpc-dev please join the hpc developers community at UAB].

Cheaha

2016-10-26T20:17:48Z

Puri@uab.edu: /* Software */

{{Main_Banner}}
'''Cheaha''' is a campus resource dedicated to enhancing research computing productivity at UAB. [http://cheaha.uabgrid.uab.edu Cheaha] is managed by [http://www.uab.edu/it UAB Information Technology's Research Computing group (UAB ITRC)] and is available to members of the UAB community in need of increased computational capacity. Cheaha supports [http://en.wikipedia.org/wiki/High-performance_computing high-performance computing (HPC)] and [http://en.wikipedia.org/wiki/High-throughput_computing high throughput computing (HTC)] paradigms.

Cheaha provides users with a traditional command-line interactive environment with access to many scientific tools that can leverage its dedicated pool of local compute resources. Alternately, users of graphical applications can start a [[Setting_Up_VNC_Session|cluster desktop]]. The local compute pool provides access to three generations of compute hardware based on the [http://en.wikipedia.org/wiki/X86_64 x86-64 64-bit architecture]. The compute resources are organized into a unified Research Computing System. The compute fabric for this system is anchored by the Cheaha cluster, [[ Resources |a commodity cluster with several generations of hardware with approximately 3000 cores]] connected by low-latency Fourteen Data Rate (FDR) and Quad Data Rate (QDR) InfiniBand networks. The compute nodes are backed by 6.6PB raw GPFS storage on DDN SFA12KX hardware, a 180TB of high performance Lustre storage on a DDN hardware and an additional 20TB available for home directories on a traditional Hitachi SAN and other ancillary services. The compute nodes combine to provide over 120TFlops of dedicated computing power.

Cheaha is composed of resources that span data centers located in the UAB Shared Computing facility UAB 936 Building and the RUST Computer Center. Resource design and development is lead by UAB IT Research Computing in open collaboration with community members. Operational [mailto:support@vo.uabgrid.uab.edu support] is provided by UAB IT's Research Computing group.

Cheaha is named in honor of [http://en.wikipedia.org/wiki/Cheaha_Mountain Cheaha Mountain], the highest peak in the state of Alabama. Cheaha is a popular destination whose summit offers clear vistas of the surrounding landscape. (Cheaha Mountain photo-streams on [http://www.flickr.com/search/?q=cheaha Flikr] and [http://picasaweb.google.com/lh/view?q=cheaha&psc=G&filter=1# Picasa]).

== Using ==

=== Getting Started ===

For information on getting an account, logging in, and running a job, please see [[Cheaha2_GettingStarted|Getting Started]].

== History ==

[[Image:Research-computing-platform.png|right|thumb|450px|Logical Diagram of Cheaha Configuration]]

=== 2005 ===

In 2002 UAB was awarded an infrastructure development grant through the NSF EPsCoR program. This led to the 2005 acquisition of a 64 node compute cluster with two AMD Opteron 242 1.6Ghz CPUs per node (128 total cores). This cluster was named Cheaha. Cheaha expanded the compute capacity available at UAB and was the first general-access resource for the community. It lead to expanded roles for UAB IT in research computing support through the development of the UAB Shared HPC Facility in BEC and provided further engagement in Globus-based grid computing resource development on campus via UABgrid and regionally via [http://www.suragrid.org SURAgrid].

=== 2008 ===

In 2008, money was allocated by UAB IT for hardware upgrades which lead to the acquisition of an additional 192 cores based on a Dell clustering solution with Intel Quad-Core E5450 3.0Ghz CPU in August of 2008. This uprade migrated Cheaha's core infrastructure to the Dell blade clustering solution. It provided a 3 fold increase in processor density over the original hardware and enables more computing power to be located in the same physical space with room for expansion, an important consideration in light of the continued growth in processing demand. This hardware represented a major technology upgrade that included space for additional expansion to address over-all capacity demand and enable resource reservation.

The 2008 upgrade began a continuous resource improvement plan that includes a phased development approach for Cheaha with on-going increases in capacity and feature enhancements being brought into production via an [http://projects.uabgrid.uab.edu/cheaha open community process].

Software improvements rolled into the 2008 upgrade included grid computing services to access distributed compute resources and orchestrate jobs using the [http://www.gridway.org GridWay] meta-scheduler. An initial 10Gigabit Ethernet link establishing the UABgrid Research Network was designed to supports high speed data transfers between clusters connected to this network.

=== 2009 ===

In 2009, annual investment funds were directed toward establishing a fully connected dual data rate Infiniband network between the compute nodes added in 2008 and laying the foundation for a research storage system with a 60TB DDN storage system accessed via the Lustre distributed file system. The Infiniband and storage fabrics were designed to support significant increases in research data sets and their associate analytical demand.

=== 2010 ===

In 2010, UAB was awarded an NIH Small Instrumentation Grant (SIG) to further increase analytical and storage capacity. The grant funds were combined with the annual investment funds adding 576 cores (48 nodes) based on the Intel Westmere 2.66 GHz CPU, a quad data rate Infiniband fabric with 32 uplinks, an additional 120 TB of storage for the DDN fabric, and additional hardware to improve reliability. Additional improvements to the research compute platform involved extending the UAB Research Network to link the BEC and RUST data centers, adding 20TB of user and ancillary services storage

=== 2012 ===

In 2012, UAB IT Research Computing invested in the foundation hardware to expand long term storage and virtual machine capabilities with aqcuisition of 12 Dell 720xd system, each containing 16 cores, 96GB RAM, and 36TB of storage, creating a 192 core and 432TB virtual compute and storage fabric.

Additionaly hardware investment by the School of Public Health's Section on Statistical Genetics added three 384GB large memory nodes and an additional 48 cores to the QDR Infiniband fabric.

=== 2013 ===

In 2013, UAB IT Research Computing acquired an [http://blogs.uabgrid.uab.edu/jpr/2013/03/were-going-with-openstack/ OpenStack cloud and Ceph storage software fabric] through a partnership between Dell and Inktank in order to extend cloud computing solutions to the researchers at UAB and enhance the interfacing capabilities for HPC. This fabric is under [http://dev.uabgrid.uab.edu active development] and will see feature releases in the coming months.

=== 2015/2016 ===

In 2015/2016 UAB IT Research computing acquired 96 2x12 core (2304 cores total) 2.5 GHz Intel Xeon E5-2680 v3 compute nodes with FDR InfiniBand interconnect. Out of the 96 compute nodes, 36 nodes have 128 GB RAM, 38 nodes have 256 GB RAM, and 14 nodes have 384 GB RAM. There are also four compute nodes with the Intel Xeon Phi 7210 accelerator cards and four compute nodes with the NVIDIA K80 GPUs. More information can be found at [[Resources]].

== Grant and Publication Resources ==

The following description may prove useful in summarizing the services available via Cheaha. If you are using Cheaha for grant funded research please send information about your grant (funding source and grant number), a statement of intent for the research project and a list of the applications you are using to UAB IT Research Computing. If you are using Cheaha for exploratory research, please send a similar note on your research interest. Finally, any publications that rely on computations performed on Cheaha should include a statement acknowledging the use of UAB Research Computing facilities in your research, see the suggested example below. Please note, your acknowledgment may also need to include an addition statement acknowledging grant-funded hardware. We also ask that you send any references to publications based on your use of Cheaha compute resources.

=== Description of Cheaha for Grants ===

UAB IT Research Computing maintains high performance compute and storage resources for investigators. The Cheaha compute cluster provides 3120 conventional CPU cores across five generations of hardware that provide over 120 TFLOP/s of combined computational performance, and 20 TB of system memory interconnected via an Infiniband network. A high-performance, 6.6PB raw GPFS storage on DDN SFA12KX hardware and 180TB Lustre parallel file system built on a Direct Data Network (DDN) hardware platform is also connected to these cores via the Infiniband fabric. An additional 20TB of traditional SAN storage and 432TB of OpenStack+Ceph storage is available via a 10+ GigE network fabric. This general access compute fabric is available to all UAB investigators.

Additionally, NIH funded investigators are granted priority access to the NIH SIG award acquired compute and storage pool that includes an additional 576 2.66GHz Intel-based compute cores, 2.3TB RAM and 180TB in the high-performance Lustre parallel file system all interconnected via a QDR Infiniband network fabric.

=== Acknowledgment in Publications ===

This work was supported in part by the research computing resources acquired and managed by UAB IT Research Computing. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the University of Alabama at Birmingham.

Additionally, NIH funded researchers using the cluster for genomics or proteomics analyses should include the following statement regarding the NIH funded gen3 (SIG) compute resources.

"Computational portions of this research were supported by NIH S10RR026723. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Institutes of Health"

== System Profile ==

=== Performance ===
{{CheahaTflops}}

=== Hardware ===

The Cheaha Compute Platform includes three generations of commodity compute hardware, totaling 868 compute cores, 2.8TB of RAM, and over 200TB of storage.

The hardware is grouped into generations designated gen1, gen2, and gen3 (oldest to newest). The following descriptions highlight the hardware profile for each generation.

* <strike> Generation 1 (gen1) -- 64 2-CPU AMD 1.6 GHz compute nodes with Gigabit interconnect. This is the original hardware collection purchased with NSF EPSCoR funds in 2005, approx $150K. These nodes are sometimes called the "Verari" nodes. These nodes are tagged as "verari-compute-#-#" in the ROCKS naming convention.</strike> (gen1 was decommissioned in June 2013)
* Generation 2 (gen2) -- 24 2x4 core (196 cores total) Intel 3.0 GHz Intel compute nodes with dual data rate Infiniband interconnect and the initial high-perf storage implementation using 60TB DDN. This is the hardware collection purchased exclusively with the annual VPIT funds allocation, approx $150K/yr for the 2008 and 2009 fiscal years. These nodes are sometimes confusingly called "cheaha2" or "cheaha" nodes. These nodes are tagged as "cheaha-compute-#-#" in the ROCKS naming convention.
* Generation 3 (gen3) -- 48 2x6 core (576 cores total) 2.66 GHz Intel compute nodes with quad data rate Infiniband, ScaleMP, and the high-perf storage build-out for capacity and redundancy with 120TB DDN. This is the hardware collection purchased with a combination of the NIH SIG funds and some of the 2010 annual VPIT investment. These nodes were given the code name "sipsey" and tagged as such in the node naming for the queue system. These nodes are tagged as "sipsey-compute-#-#" in the ROCKS naming convention. 16 of the gen3 nodes (sipsey-compute-0-1 thru sipsey-compute-0-16) were upgraded in 2014 from 48GB to 96GB of memory per node.
* Generation 4 (gen4) -- 3 16 core (48 cores total) compute nodes. This hardware collection purchase by [http://www.soph.uab.edu/ssg/people/tiwari Dr. Hemant Tiwari of SSG]. These nodes were given the code name "ssg" and tagged as such in the node naming for the queue system. These nodes are tagged as "ssg-compute-0-#" in the ROCKS naming convention.
* Generation 6 (gen6) --
** 44 Compute Nodes with two 12 core processors (Intel Xeon E5-2680 v3 2.5GHz) with 128GB DDR4 RAM, FDR InfiniBand and 10GigE network cards (4 nodes with NVIDIA K80 GPUs and 4 nodes with Intel Xeon Phi 7120P accelerators)
** 38 Compute Nodes with two 12 core processors (Intel Xeon E5-2680 v3 2.5GHz) with 256GB DDR4 RAM, FDR InfiniBand and 10GigE network cards
** 14 Compute Nodes with two 12 core processors (Intel Xeon E5-2680 v3 2.5GHz) with 384GB DDR4 RAM, FDR InfiniBand and 10GigE network card
** FDR InfiniBand Switch
** 10Gigabit Ethernet Switch
** Management node and gigabit switch for cluster management
** Bright Advanced Cluster Management software licenses

Summarized, Cheaha's compute pool includes:
* gen4 is 48 cores of [http://ark.intel.com/products/64583/Intel-Xeon-Processor-E5-2680-20M-Cache-2_70-GHz-8_00-GTs-Intel-QPI 2.70GHz eight-core Intel Xeon E5-2680 processors] with 24G of RAM per core or 384GB total
* gen3 is 192 cores of [http://ark.intel.com/products/47922/Intel-Xeon-Processor-X5650-12M-Cache-2_66-GHz-6_40-GTs-Intel-QPI?q=x5650 2.67GHz six-core Intel Xeon X5650 processors] with 8Gb RAM per core or 96GB total
* gen3 is 384 cores of [http://ark.intel.com/products/47922/Intel-Xeon-Processor-X5650-12M-Cache-2_66-GHz-6_40-GTs-Intel-QPI?q=x5650 2.67GHz six-core Intel Xeon X5650 processors] with 4Gb RAM per core or 48GB total
* gen2 is 192 cores of [http://ark.intel.com/products/33083/Intel-Xeon-Processor-E5450-12M-Cache-3_00-GHz-1333-MHz-FSB 3.0GHz quad-core Intel Xeon E5450 processors] with 2Gb RAM per core
* <strike> gen1 is 100 cores of 1.6GhZ AMD Opteron 242 processors with 1Gb RAM per core </strike> (decomissioned June 2013)

{|border="1" cellpadding="2" cellspacing="0"
|+ Physical Nodes
|- bgcolor=grey
!gen!!queue!!#nodes!!cores/node!!RAM/node
|-
|gen6|| default || 44 || 24 || 128G
|-
|gen6|| default || 38 || 24 || 256G
|-
|gen6|| default || 14 || 24 || 384G
|-
|gen5||Ceph/OpenStack|| 12 || 20 || 96G
|-
|gen4||ssg||3||16||385G
|-
|gen3||sipsey||16||12||96G
|-
|gen3||sipsey||32||12||48G
|-
|gen2||cheaha||24||8||16G
|}

=== Software ===

Details of the software available on Cheaha can be found on the [http://me.eng.uab.edu/wiki/index.php?title=Cheaha#Installed_software Cheaha cluster configuration page], an overview follows.

Cheaha uses [http://modules.sourceforge.net/ Environment Modules] to support account configuration. Please follow these [http://me.eng.uab.edu/wiki/index.php?title=Cheaha#Environment_Modules specific steps for using environment modules].

Cheaha's software stack is built with the [[Slurm]] cluster manager and job scheduling system. Cheaha's operating system is CentOS with the following major cluster components:
* BrightCM 7.2
* CentOS 7.2 x86_64
* [[Slurm]] 15.08

A brief summary of the some of the available computational software and tools available includes:
* Amber
* FFTW
* Gromacs
* GSL
* NAMD
* VMD
* Intel Compilers
* GNU Compilers
* Java
* R
* OpenMPI
* MATLAB

=== Network ===

Cheaha is connected to the UAB Research Network which provides a dedicated 10Gbs networking backplane between clusters located in the 936 data center and the campus network core. Data transfers rates of almost 8Gbps between these hosts have been demonstrated using Grid FTP, a multi-channel file transfer service that is used to move data between clusters as part of the job management operations. This performance promises very efficient job management and the seamless integration of other clusters as connectivity to the research network is expanded.

=== Benchmarks ===

The continuous resource improvement process involves collecting benchmarks of the system. One of the measures of greatest interest to users of the system are benchmarks of specific application codes. The following benchmarks have been performed on the system and will be further expanded as additional benchmarks are performed.

* [[Cheaha-BGL_Comparison|Cheaha-BGL Comparison]]

* [[Gromacs_Benchmark|Gromacs]]

* [[NAMD_Benchmarks|NAMD]]

=== Performance Statistics ===

Cheaha uses Ganglia to report cluster performance data. This information provides a helpful overview of the current and historical operating stats for Cheaha. You can access the Ganglia monitoring page [https://cheaha.uabgrid.uab.edu/ganglia/ here].

== Availability ==

Cheaha is a general-purpose computer resource made available to the UAB community by UAB IT. As such, it is available for legitimate research and educational needs and is governed by [http://www.uabgrid.uab.edu/aup UAB's Acceptable Use Policy (AUP)] for computer resources.

Many software packages commonly used across UAB are available via Cheaha.

To request access to Cheaha, please send a request to [mailto:support@vo.uabgrid.uab.edu send a request] to the cluster support group.

Cheaha's intended use implies broad access to the community, however, no guarantees are made that specific computational resources will be available to all users. Availability guarantees can only be made for reserved resources.

=== Secure Shell Access ===

Please configure you client secure shell software to use the official host name to access Cheaha:

<pre>
cheaha.rc.uab.edu
</pre>

== Scheduling Framework ==

[http://slurm.schedmd.com/ Slurm] is a queue management system and stands for Simple Linux Utility for Resource Management. Slurm was developed at the Lawrence Livermore National Lab and currently runs some of the largest compute clusters in the world. '''[[Slurm]]''' is now the primary job manager on Cheaha, it replaces SUN Grid Engine (SGE) the job manager used earlier.

Slurm is similar in many ways to GridEngine or most other queue systems. You write a batch script then submit it to the queue manager (scheduler). The queue manager then schedules your job to run on the queue (or '''partition''' in Slurm parlance) that you designate. Below we will provide an outline of how to submit jobs to Slurm, how Slurm decides when to schedule your job, and how to monitor progress.

== Support ==

Operational support for Cheaha is provided by the Research Computing group in UAB IT. For questions regarding the operational status of Cheaha please send your request to [mailto:support@vo.uabgrid.uab.edu support@vo.uabgrid.uab.edu]. As a user of Cheaha you will automatically by subscribed to the hpc-announce email list. This subscription is mandatory for all users of Cheaha. It is our way of communicating important information regarding Cheaha to you. The traffic on this list is restricted to official communication and has a very low volume.

We have limited capacity, however, to support non-operational issue like "How do I write a job script" or "How do I compile a program". For such requests, you may find it more fruitful to send your questions to the hpc-users email list and request help from our peers in the HPC community at UAB. As with all mailing lists, please observe [http://lifehacker.com/5473859/basic-etiquette-for-email-lists-and-forums common mailing etiquette].

Finally, please remember that as you learned about HPC from others it becomes part of your responsibilty to help others on their quest. You should update this documentation or respond to mailing list requests of others.

You can subscribe to hpc-users by sending an email to:

[mailto:sympa@vo.uabgrid.uab.edu?subject=subscribe%20hpc-users sympa@vo.uabgrid.uab.edu with the subject ''subscribe hpc-users''].

You can unsubribe from hpc-users by sending an email to:

[mailto:sympa@vo.uabgrid.uab.edu?subject=unsubscribe%20hpc-users sympa@vo.uabgrid.uab.edu with the subject ''unsubscribe hpc-users''].

You can review archives of the list in the [http://vo.uabgrid.uab.edu/sympa/arc/hpc-users web hpc-archives].

If you need help using the list service please send an email to:

[mailto:sympa@vo.uabgrid.uab.edu?subject=help sympa@vo.uabgrid.uab.edu with the subject ''help'']

If you have questions about the operation of the list itself, please send an email to the owners of the list:

[mailto:hpc-users-request@vo.uabgrid.uab.edu sympa@vo.uabgrid.uab.edu with a subject relavent to your issue with the list]

If you are interested in contributing to the enhancement of HPC features at UAB or would like to talk to other cluster administrators, [mailto:sympa@vo.uabgrid.uab.edu?subject=subscribe%20hpc-dev please join the hpc developers community at UAB].