Cheaha - User contributions [en]

OOD IGV

2019-10-16T19:45:35Z

Curtish@uab.edu:

== First time setup ==

# get a cheaha account (see [[Cheaha_GettingStarted]])
* then install EITHER
* via [[#Install_IGV_via_Job|OOD job launcher]]
* OR
* via [[#Install_via_Terminal_in_OOD_Desktop|Terminal on the OOD Desktop]]

== Install IGV via Job ==
# launch Job Composer/Create New Job/From a Specified Path: https://rc.uab.edu/pun/sys/myjobs/new_from_path and setup the job
* Source path: '''/share/apps/ngs-ccts/ood-igv/jobs'''
* Name: '''setup IGV 2.5'''
* Script Name: '''2.5.sh'''
* Cluster: '''Cheaha'''
* '''SAVE'''
* [[File:A1.ood job composer.jpg|700px]]
# Run/Submit the job
* Click on the green "play" arrow. [[File:A2.ood job submit.png|700px]]
* Status changes to "queued" [[File:A3.ood job queued.png|700px]]
* wait until job completes. [[File:A4.ood job completed.png|700px]]
# now open OOD desktop to launch IGV from desktop icon
* see [[#Running_IGV_from_OOD_Desktop]]

== Install via Terminal in OOD Desktop ==
# launch an interactive desktop with OOD https://rc.uab.edu
* Requst an OOD Desktop [[File:A.ood start desktop.png|700px]]
* Set Request RAM and HOURS [[File:A.ood set mem.png|700px]]
* Open the desktop, once running [[File:A.ood launch desktop.png|700px]]
* desktop open [[File:A.ood desktop.png|700px]]
# start a "Terminal"
* open terminal app [[File:B.ood with terminal highlight.jpg|700px]]
# in terminal, enter <nowiki>/share/apps/ngs-ccts/ood-igv/2.5.sh</nowiki>
* enter path to installer [[File:C.ood terminal setup 25.png|700px]]
# that will install IGV locally, and launch it.
* installer will scroll a lot of text, some in alarming colors.
* a few seconds after the text stop, the desktop icon and loading bar will appear
* installer finished, IGV loading [[File:D.ood setup ivg loading.png|700px]]

== Running IGV from OOD Desktop ==

# Setup should create a desktop icon called "IGV-2.5.sh"
* [[File:G.ood desktop with icon.png|700px]]
# In the future, you can just start OOD, then click on "IGV-2.5.sh"
* [[File:E.ood igv loading.png|800px]]

== Script source code ==
Code can be found at https://gitlab.rc.uab.edu/CCTS-Informatics-Pipelines/ood-igv

File:G.ood desktop with icon.png

2019-10-16T19:41:37Z

Curtish@uab.edu: Curtish@uab.edu uploaded a new version of "File:G.ood desktop with icon.png"

OOD IGV

2019-10-16T19:33:04Z

Curtish@uab.edu:

== First time setup ==

# get a cheaha account (see [[Cheaha_GettingStarted]])

== Install IGV via Job ==
# launch Job Composer/Create New Job/From a Specified Path: https://rc.uab.edu/pun/sys/myjobs/new_from_path and setup the job
* Source path: '''/share/apps/ngs-ccts/ood-igv/jobs'''
* Name: '''setup IGV 2.5'''
* Script Name: '''2.5.sh'''
* Cluster: '''Cheaha'''
* '''SAVE'''
* [[File:A1.ood job composer.jpg|700px]]
# Run/Submit the job
* Click on the green "play" arrow. [[File:A2.ood job submit.png|700px]]
* Status changes to "queued" [[File:A3.ood job queued.png|700px]]
* wait until job completes. [[File:A4.ood job completed.png|700px]]
# now open OOD desktop to launch IGV from desktop icon
* see [[#Running_IGV_from_OOD_Desktop]]

== Install via Terminal in OOD Desktop ==
# launch an interactive desktop with OOD https://rc.uab.edu
* Requst an OOD Desktop [[File:A.ood start desktop.png|700px]]
* Set Request RAM and HOURS [[File:A.ood set mem.png|700px]]
* Open the desktop, once running [[File:A.ood launch desktop.png|700px]]
* desktop open [[File:A.ood desktop.png|700px]]
# start a "Terminal"
* open terminal app [[File:B.ood with terminal highlight.jpg|700px]]
# in terminal, enter <nowiki>/share/apps/ngs-ccts/ood-igv/2.5.sh</nowiki>
* enter path to installer [[File:C.ood terminal setup 25.png|700px]]
# that will install IGV locally, and launch it.
* installer will scroll a lot of text, some in alarming colors.
* a few seconds after the text stop, the desktop icon and loading bar will appear
* installer finished, IGV loading [[File:D.ood setup ivg loading.png|700px]]

== Running IGV from OOD Desktop ==

# Setup should create a desktop icon called "IGV-2.5.sh"
* [[File:G.ood desktop with icon.png|700px]]
# In the future, you can just start OOD, then click on "IGV-2.5.sh"
* [[File:E.ood igv loading.png|800px]]

== Script source code ==
Code can be found at https://gitlab.rc.uab.edu/CCTS-Informatics-Pipelines/ood-igv

File:G.ood desktop with icon.png

2019-10-16T19:32:03Z

Curtish@uab.edu: Curtish@uab.edu uploaded a new version of "File:G.ood desktop with icon.png": highlight icon

OOD IGV

2019-10-16T19:28:37Z

Curtish@uab.edu:

== First time setup ==

# get a cheaha account (see [[Cheaha_GettingStarted]])

== Install IGV via Job ==
# launch Job Composer/Create New Job/From a Specified Path: https://rc.uab.edu/pun/sys/myjobs/new_from_path and setup the job
* Source path: '''/share/apps/ngs-ccts/ood-igv/jobs'''
* Name: '''setup IGV 2.5'''
* Script Name: '''2.5.sh'''
* Cluster: '''Cheaha'''
* '''SAVE'''
* [[File:A1.ood job composer.jpg|500px]]
# Run/Submit the job
* Click on the green "play" arrow. [[File:A2.ood job submit.png|500px]]
* Status changes to "queued" [[File:A3.ood job queued.png|500px]]
* wait until job completes. [[File:A4.ood job completed.png|500px]]
# now open OOD desktop to launch IGV from desktop icon
* see [[OOD_IGV#]]

== Install via Terminal in OOD Desktop ==
# launch an interactive desktop with OOD https://rc.uab.edu
* Requst an OOD Desktop [[File:A.ood start desktop.png|700px]]
* Set Request RAM and HOURS [[File:A.ood set mem.png|700px]]
* Open the desktop, once running [[File:A.ood launch desktop.png|700px]]
* desktop open [[File:A.ood desktop.png|700px]]
# start a "Terminal"
* open terminal app [[File:B.ood with terminal highlight.jpg|700px]]
# in terminal, enter <nowiki>/share/apps/ngs-ccts/ood-igv/2.5.sh</nowiki>
* enter path to installer [[File:C.ood terminal setup 25.png|700px]]
# that will install IGV locally, and launch it.
* installer will scroll a lot of text, some in alarming colors.
* a few seconds after the text stop, the desktop icon and loading bar will appear
* installer finished, IGV loading [[File:D.ood setup ivg loading.png|700px]]

== Running IGV from OOD Desktop ==

# Setup should create a desktop icon called "IGV-2.5.sh"
* [[File:G.ood desktop with icon.png|700px]]
# In the future, you can just start OOD, then click on "IGV-2.5.sh"
* [[File:E.ood igv loading.png|800px]]

== Script source code ==
Code can be found at https://gitlab.rc.uab.edu/CCTS-Informatics-Pipelines/ood-igv

File:A1.ood job composer.jpg

2019-10-16T19:26:32Z

Curtish@uab.edu:

File:A4.ood job completed.png

2019-10-16T19:20:17Z

Curtish@uab.edu:

File:A3.ood job queued.png

2019-10-16T19:20:03Z

Curtish@uab.edu:

File:A2.ood job submit.png

2019-10-16T19:19:49Z

Curtish@uab.edu:

File:A1.ood job composer.GIF

2019-10-16T19:19:37Z

Curtish@uab.edu:

OOD IGV

2019-10-15T20:11:57Z

Curtish@uab.edu: add graffiti around terminal start icon

== First time setup ==

# get a cheaha account (see [[Cheaha_GettingStarted]])
# launch an interactive desktop with OOD https://rc.uab.edu
* Requst an OOD Desktop [[File:A.ood start desktop.png|700px]]
* Set Request RAM and HOURS [[File:A.ood set mem.png|700px]]
* Open the desktop, once running [[File:A.ood launch desktop.png|700px]]
* desktop open [[File:A.ood desktop.png|700px]]
# start a "Terminal"
* open terminal app [[File:B.ood with terminal highlight.jpg|700px]]
# in terminal, enter <nowiki>/share/apps/ngs-ccts/ood-igv/2.5.sh</nowiki>
* enter path to installer [[File:C.ood terminal setup 25.png|700px]]
# that will install IGV locally, and launch it.
* installer will scroll a lot of text, some in alarming colors.
* a few seconds after the text stop, the desktop icon and loading bar will appear
* installer finished, IGV loading [[File:D.ood setup ivg loading.png|700px]]

== Running later ==
# Setup should create a desktop icon called "IGV-2.5.sh"
* [[File:G.ood desktop with icon.png|700px]]
# In the future, you can just start OOD, then click on "IGV-2.5.sh"
* [[File:E.ood igv loading.png|800px]]

== Script source code ==
Code can be found at https://gitlab.rc.uab.edu/CCTS-Informatics-Pipelines/ood-igv

File:B.ood with terminal highlight.jpg

2019-10-15T20:10:52Z

Curtish@uab.edu:

OOD IGV

2019-10-15T19:48:47Z

Curtish@uab.edu: add images

== First time setup ==

# get a cheaha account (see [[Cheaha_GettingStarted]])
# launch an interactive desktop with OOD https://rc.uab.edu
* Requst an OOD Desktop [[File:A.ood start desktop.png|700px]]
* Set Request RAM and HOURS [[File:A.ood set mem.png|700px]]
* Open the desktop, once running [[File:A.ood launch desktop.png|700px]]
* desktop open [[File:A.ood desktop.png|700px]]
# start a "Terminal"
* open terminal app [[File:B.ood with terminal.png|700px]]
# in terminal, enter <nowiki>/share/apps/ngs-ccts/ood-igv/2.5.sh</nowiki>
* enter path to installer [[File:C.ood terminal setup 25.png|700px]]
# that will install IGV locally, and launch it.
* installer will scroll a lot of text, some in alarming colors.
* a few seconds after the text stop, the desktop icon and loading bar will appear
* installer finished, IGV loading [[File:D.ood setup ivg loading.png|700px]]

== Running later ==
# Setup should create a desktop icon called "IGV-2.5.sh"
* [[File:G.ood desktop with icon.png|700px]]
# In the future, you can just start OOD, then click on "IGV-2.5.sh"
* [[File:E.ood igv loading.png|800px]]

== Script source code ==
Code can be found at https://gitlab.rc.uab.edu/CCTS-Informatics-Pipelines/ood-igv

OOD IGV

2019-10-15T19:27:55Z

Curtish@uab.edu:

== First time setup ==

# get a cheaha account (see [[Cheaha_GettingStarted]])
# launch an interactive desktop with OOD https://rc.uab.edu
<gallery>
File:A.ood start desktop.png|OOD Start Desktop Menu
File:A.ood set mem.png|OOD Set Desktop memory/run-time
File:A.ood launch desktop.png|Launch Desktop once it's running
</gallery>
# start a "Terminal"
# in terminal, enter <nowiki>/share/apps/ngs-ccts/ood-igv/2.5.sh</nowiki>
# that will install IGV locally, and launch it.

== Running later ==
# Setup should create a desktop icon called "IGV-2.5.sh"
# In the future, you can just start OOD, then click on "IGV-2.5.sh"

== Script source code ==
Code can be found at https://gitlab.rc.uab.edu/CCTS-Informatics-Pipelines/ood-igv

File:A.ood desktop.png

2019-10-15T19:24:24Z

Curtish@uab.edu:

File:B.ood with terminal.png

2019-10-15T19:24:08Z

Curtish@uab.edu:

File:C.ood terminal setup 25.png

2019-10-15T19:23:55Z

Curtish@uab.edu:

File:G.ood desktop with icon.png

2019-10-15T19:23:40Z

Curtish@uab.edu:

File:F.ood igv up.png

2019-10-15T19:23:29Z

Curtish@uab.edu:

File:E.ood igv loading.png

2019-10-15T19:23:09Z

Curtish@uab.edu:

File:D.ood setup ivg loading.png

2019-10-15T19:22:52Z

Curtish@uab.edu:

File:A.ood start desktop.png

2019-10-15T19:22:38Z

Curtish@uab.edu:

File:A.ood set mem.png

2019-10-15T19:22:04Z

Curtish@uab.edu:

File:A.ood launch desktop.png

2019-10-15T19:21:41Z

Curtish@uab.edu:

OOD IGV

2019-10-15T16:51:15Z

Curtish@uab.edu:

== First time setup ==

# get a cheaha account (see [[Cheaha_GettingStarted]])
# launch an interactive desktop with OOD https://rc.uab.edu
# start a "Terminal"
# in terminal, enter <nowiki>/share/apps/ngs-ccts/ood-igv/2.5.sh</nowiki>
# that will install IGV locally, and launch it.

== Running later ==
# Setup should create a desktop icon called "IGV-2.5.sh"
# In the future, you can just start OOD, then click on "IGV-2.5.sh"

== Script source code ==
Code can be found at https://gitlab.rc.uab.edu/CCTS-Informatics-Pipelines/ood-igv

OOD IGV

2019-10-15T16:47:39Z

Curtish@uab.edu:

== First time setup ==

# get a cheaha account (see [[Cheaha_GettingStarted]])
# launch an interactive desktop with OOD https://rc.uab.edu
# start a "Terminal"
# in terminal, enter <nowiki>/share/apps/ngs-ccts/ood-igv/2.5.sh</nowiki>
# that will install IGV locally, and launch it.
# That should also create a desktop icon called "IGV-2.5.sh"
# In the future, you can just start OOD, then click on "IGV-2.5.sh"

OOD IGV

2019-10-14T21:27:18Z

Curtish@uab.edu: Created page with "== First time setup == # get a cheaha account (see [Cheaha_GettingStarted]) # launch an interactive desktop with OOD https://rc.uab.edu # start a "Terminal" # in terminal,..."

== First time setup ==

# get a cheaha account (see [Cheaha_GettingStarted])
# launch an interactive desktop with OOD https://rc.uab.edu
# start a "Terminal"
# in terminal, enter
<nowiki>/share/apps/ngs-ccts/IGV/setup/IGV.sh</nowiki>

That should create a desktop icon called "LAUNCH_IGV.sh", and it should launch IGV as well.

In the future, you can just start OOD, then click on "LAUNCH_IGV"

Box.com bulk transfer

2018-04-11T01:52:54Z

Curtish@uab.edu: add .netrc info

UAB has an Enterprise contract with Box.com, which is currently in BETA.

This page describes what we have learned about doing bulk-transfers of data.

== Warning: size limitations ==

Box.com claims to have a '''''5G max filesize''''' limit
* Now '''''15G max filesize''''' (2/18/016; private email), but "that file size limit is still considered to be in a beta phase"

There was a rumor this would be increased in early 2016.
Offically, this is all we know: https://community.box.com/t5/Managing-Your-Content/What-s-the-maximum-file-size-I-can-upload/ta-p/307

If you need to work around this, you can use the Linux "[http://ss64.com/bash/split.html split]" utility

{| class="wikitable"
|-
| # chop file into 4G pieces 
split \ 
--bytes=4000m \ 
big_file.fastq.gz \ 
big_file.fastq.gz.split4g. 
|-
| # record checksums of original and chunks 
md5sum \ 
big_file.fastq.gz \ 
big_file.fastq.gz.split4g.* \ 
> big_file.fastq.gz.md5
|}

== Warning: time stamps ==

When using an FTP client to transfer data up, it is easy to lose both modification and creation timestamps. In particular, many clients will (optionally) preserve modification time, but few will (optionally) preserve creation date.

{| class="wikitable" border="1"
|-
! FTP client
! cost
! platform
! preserve mod_date
! preserve create_date
|-
| lftp
| free
| linux/cmd_line
| yes
| no
|-
| SmartFTP
| $$
| Win Only/GUI
| yes
| can be enabled
|-
| fileZilla
| free
| linux & win/GUI_only
| can be enabled
| [https://trac.filezilla-project.org/ticket/2347 no]
|-
| ftp_ssl
| free
| linux/cmd_line
| yes
| no
|}

Filezilla on create times
* request closed, no plans to fix: https://trac.filezilla-project.org/ticket/2347

== Warning: Shared-to-you folders can't be moved ==

If someone creates a folder and shares it to you (as an reader, editor, co-owner, etc), it will live in your top level directory, and you will NOT be able to move it into any subfolder.

"Currently, users can't rearrange their own view of folders they are invited to collaborate within. As you note, when someone invites you to collaborate in a folder that you have never had access to before, you will see that folder on your root level." [box.com]

'''Workaround''': if your collaborator makes you the full owner of the folder, then you will be able to move it.

'''Fix timeframe''': "We've heard requests that people be able to rearrange their views before, and this is being considered as part of a larger product experience change next year" [box.com]

== lftp mirror -R examples (UPload) ==

lftp mirror
* "mirror" copies directory hierarchies DOWN from box.com to local
* "mirror -R" copies directory hierarchies UP from local to box.com

error handling
* the box server frequently looses connection (fails) on particular files
* just re-run the "mirror -R" and it will upload only new/failed files..

=== Interactive lftp ===
{| class="wikitable" border="1"
|-
| lftp ftp.box.com 
> user ''BLAZERID''@uab.edu ''users_BOX_external_password'' 
> mirror --parallel=10 -R ''local_src_dir'' ''box_dest_dir''
|}

=== Single-line lftp (non-shared box) ===

'''Warning''': This one makes the password visible to "ps" so should only be used on personal machines
{| class="wikitable" border="1"
|-
| lftp -u ''BLAZERID''@uab.edu,''users_BOX_external_password'' ftp.box.com << EOF 
mirror --parallel=10 -R ''local_src_dir'' ''box_dest_dir'' 
EOF
|}

=== scripted lftp ===
{| class="wikitable" border="1"
|-
| cat > box_upload.lftp << EOF 
open ftp.box.com 
user ''BLAZERID''@uab.edu ''users_BOX_external_password'' 
mirror --parallel=10 -R ''local_src_dir'' /''box_dest_dir'' 
EOF 
chmod 700 box_upload.lftp 
lftp -f box_upload.lftp ; echo lftp_RC=$?
|}

=== scripted lftp - externalize password ===

file ~/.netrc contains your box external password once (works for wget, lftp, etc)
{| class="wikitable" border="1"
|-
| cat >> ~/.netrc << EOF 
machine ftp.box.com 
login ''BLAZERID''@uab.edu 
password "user_Box_External_PW" 
 
EOF 
 
chmod 700 ~/.netrc 
|}

then for each transfer, you create a local .lftp file w/o a password.
Much more secure and easy to keep up to date.
{| class="wikitable" border="1"
|-
| cat > box_upload.lftp << EOF 
open ftp.box.com 
mirror -R ''local_src_dir'' /''box_dest_dir'' 
EOF 
 
lftp -f box_upload.lftp ; echo lftp_RC=$?
|}

== lftp mirror examples (DOWNload) ==

=== scripted lftp ===

Arguments
* --loop keep restarting until there are no new files left to download - helps if someone else is uploading to that directory while you're downloading it!
* -v verbose level 1: includes bytes transferred and transfer speed.
* --parallel=10 use 10 concurrent TCP/IP connections (much faster)
{| class="wikitable" border="1"
|-
| cat > box_download.lftp << EOF 
open ftp.box.com 
user ''BLAZERID''@uab.edu ''users_BOX_external_password'' 
mirror --loop -v --parallel=10 /''box_remote_src_dir'' ''local_dest_dir'' 
EOF 
chmod 700 box_download.lftp 
lftp -f box_download.lftp ; echo lftp_RC=$?
|}

== Issues to resolve ==

* Routing over Iternet2
* we see our traffic randomly going over the commodity internet

== Linux Support ==
Unfortunately, Box doesn't provide a Linux client (is it on the road map?).

Dmtcp Checkpointing

2017-09-14T19:21:10Z

Curtish@uab.edu: add keyword checkpoint

'''DMTCP: Distributed MultiThreaded CheckPointing'''

Search keywords: checkpoint check point

http://dmtcp.sourceforge.net/FAQ.html

Available on Cheaha.RC
* module load DMTCP/2.4.5
* module load DMTCP/2.5.0

example [[Slurm|SLURM]] Job: https://github.com/dmtcp/dmtcp/tree/master/plugin/batch-queue/job_examples
* to use "srun" or not? - just for better reporting via sacct -j ### / sstat -j ###

Dmtcp

2017-09-14T19:17:33Z

Curtish@uab.edu: Curtish@uab.edu moved page Dmtcp to Dmtcp Checkpointing: make this page searchable under the keyword checkpoint

#REDIRECT [[Dmtcp Checkpointing]]

Dmtcp Checkpointing

2017-09-14T19:17:33Z

Curtish@uab.edu: Curtish@uab.edu moved page Dmtcp to Dmtcp Checkpointing: make this page searchable under the keyword checkpoint

'''DMTCP: Distributed MultiThreaded CheckPointing'''

http://dmtcp.sourceforge.net/FAQ.html

Available on Cheaha.RC
* module load DMTCP/2.4.5
* module load DMTCP/2.5.0

example [[Slurm|SLURM]] Job: https://github.com/dmtcp/dmtcp/tree/master/plugin/batch-queue/job_examples
* to use "srun" or not? - just for better reporting via sacct -j ### / sstat -j ###

Dmtcp Checkpointing

2017-02-17T20:11:08Z

Curtish@uab.edu: add module load

Dmtcp Checkpointing

2017-02-07T16:56:33Z

Curtish@uab.edu:

'''DMTCP: Distributed MultiThreaded CheckPointing'''

Coming soon to Cheaha.RC as a module: http://dmtcp.sourceforge.net/FAQ.html

example [[Slurm|SLURM]] Job: https://github.com/dmtcp/dmtcp/tree/master/plugin/batch-queue/job_examples
* to use "srun" or not? - just for better reporting via sacct -j ### / sstat -j ###

Dmtcp Checkpointing

2017-02-07T16:46:45Z

Curtish@uab.edu: add link to slurm page

Dmtcp Checkpointing

2017-02-07T16:43:15Z

Curtish@uab.edu: created based on discussion with Thomas Anthony

'''DMTCP: Distributed MultiThreaded CheckPointing'''

Coming soon to Cheaha.RC as a module: http://dmtcp.sourceforge.net/FAQ.html

example SLURM Job: https://github.com/dmtcp/dmtcp/tree/master/plugin/batch-queue/job_examples

Slurm

2016-03-11T19:56:47Z

Curtish@uab.edu: /* 20160311 Mike H */

[http://slurm.schedmd.com/ SLURM] is a queue management system and stands for Simple Linux Utility for Resource Management. SLURM was developed at the Lawrence Livermore National Lab and currently runs some of the largest compute clusters in the world. SLURM is the primary job manager on Cheaha (BigGreen- new hardware) while GridEngine continues to be the job manager on the old hardware.

SLURM is similar in many ways to GridEngine or most other queue systems. You write a batch script then submit it to the queue manager (scheduler). The queue manager then schedules your job to run on the queue (or '''partition''' in SLURM parlance) that you designate. Below we will provide an outline of how to submit jobs to SLURM, how SLURM decides when to schedule your job and how to monitor progress.

=== General SLURM Documentation ===
The primary source for documentation on SLURM usage and commands can be found at the [http://slurm.schedmd.com/ SLURM] site. If you Google for SLURM questions, you'll often see the Lawrence Livermore pages as the top hits, but these tend to be outdated.

A great way to get details on the SLURM commands is the man pages available from the Cheaha cluster. For example, if you type the following command:

<pre>
man sbatch
</pre>
you'll get the manual page for the sbatch command.

=== Logging on and Running Jobs from the command line ===
Once you've gone through the [https://docs.uabgrid.uab.edu/wiki/Cheaha_GettingStarted#Access_.28Cluster_Account_Request.29 account setup procedure] and obtained a suitable [https://docs.uabgrid.uab.edu/wiki/Cheaha_GettingStarted#Client_Configuration terminal application], you can login to the Cheaha system via ssh

ssh '''blazerid'''@cheaha.rc.uab.edu

Cheaha (new hardware) run the CentOS 7 version of the Linux operating system and commands are run under the "bash" shell. There are a number of Linux and [http://www.gnu.org/software/bash/manual/bashref.html bash references], [http://cli.learncodethehardway.org/bash_cheat_sheet.pdf cheat sheets] and [http://www.tldp.org/LDP/Bash-Beginners-Guide/html/ tutorials] available on the web.

=== Typical Workflow ===
* Stage data to $USER_SCRATCH (your scratch directory)
* Research how to run your code in "batch" mode. Batch mode typically means the ability to run it from the command line without requiring any interaction from the user.
* Identify the appropriate resources needed to run the job. The following are mandatory resource requests for all jobs on Cheaha
** Number of processor cores required by the job
** Maximum memory (RAM) required per core
** Maximum runtime
* Write a job script specifying queuing system parameters, resource requests and commands to run program
* Submit script to queuing system (sbatch script.job)
* Monitor job (squeue)
* Review the results and resubmit as necessary
* Clean up the scratch directory by moving or deleting the data off of the cluster

=== Interactive Session ===
Head Node (The command-line interface after you login to Cheaha ) is supposed to be used for submitting jobs and/or lighter prep work required for the job scripts. '''You are not supposed to run heavy computations on the head node'''. If you have a heavier workload to prepare for a batch job (eg. compiling code or other manipulations of data) or your compute application requires interactive control, you should request a dedicated interactive node for this work.

Interactive resources are requested by submitting an "interactive" job to the scheduler. Interactive jobs will provide you a command line on a compute resource that you can use just like you would the command line on the head node. The difference is that the scheduler has dedicated the requested resources to your job and you can run your interactive commands without having to worry about impacting other users on the head node.
Interactive jobs are requested with the '''srun''' command

<pre>
srun -n 1 -N 1 -t 01:00:00 --pty /bin/bash
</pre>

This command requests for 1 core (-n) on 1 node (-N) for 1 hour (-t).

More advanced interactive scenarios to support graphical applications are available using [https://docs.uabgrid.uab.edu/wiki/Setting_Up_VNC_Session VNC] or X11 tunneling [http://www.uab.edu/it/software X-Win32 2014 for Windows]
== Updates ==

=== 20160311 partitions & graphical interactive ===

Howdy, the new changes are in place. The primary focus of the changes were to:
# Change the scheduling algorithm to one that allows jobs to share compute nodes (i.e. Slurm will allocate CPU cores now instead of complete compute nodes).
#We added partitions (in SGE they were called queues) with the following characteristics (these may change over time as we tweak things):
#* short (default partition): Priority 2 :: Max Runtime 2 hours
#* medium: Priority 4 :: Max Runtime 50 hours
#* long: Priority 6 :: Max Runtime 159 hours (6 days 6 hours)
#* interactive: Priority 10 :: Max Runtime 2 hours
# In order to run a job in a partition other than "short" you'll need to specifically request it using the --partition argument (--time=48:00:00 --partition=medium)
# Graphical interactive jobs now work. You can run an interactive job using the sinteractive command, for example:
sinteractive --time=00:05:00 --job-name=sinteractiveTest --ntasks=1 --mem=1024

More to come and please let us know of any issues or concerns.

Slurm

2016-03-11T19:56:25Z

Curtish@uab.edu: updates - graphical interactive works

[http://slurm.schedmd.com/ SLURM] is a queue management system and stands for Simple Linux Utility for Resource Management. SLURM was developed at the Lawrence Livermore National Lab and currently runs some of the largest compute clusters in the world. SLURM is the primary job manager on Cheaha (BigGreen- new hardware) while GridEngine continues to be the job manager on the old hardware.

SLURM is similar in many ways to GridEngine or most other queue systems. You write a batch script then submit it to the queue manager (scheduler). The queue manager then schedules your job to run on the queue (or '''partition''' in SLURM parlance) that you designate. Below we will provide an outline of how to submit jobs to SLURM, how SLURM decides when to schedule your job and how to monitor progress.

=== General SLURM Documentation ===
The primary source for documentation on SLURM usage and commands can be found at the [http://slurm.schedmd.com/ SLURM] site. If you Google for SLURM questions, you'll often see the Lawrence Livermore pages as the top hits, but these tend to be outdated.

A great way to get details on the SLURM commands is the man pages available from the Cheaha cluster. For example, if you type the following command:

<pre>
man sbatch
</pre>
you'll get the manual page for the sbatch command.

=== Logging on and Running Jobs from the command line ===
Once you've gone through the [https://docs.uabgrid.uab.edu/wiki/Cheaha_GettingStarted#Access_.28Cluster_Account_Request.29 account setup procedure] and obtained a suitable [https://docs.uabgrid.uab.edu/wiki/Cheaha_GettingStarted#Client_Configuration terminal application], you can login to the Cheaha system via ssh

ssh '''blazerid'''@cheaha.rc.uab.edu

Cheaha (new hardware) run the CentOS 7 version of the Linux operating system and commands are run under the "bash" shell. There are a number of Linux and [http://www.gnu.org/software/bash/manual/bashref.html bash references], [http://cli.learncodethehardway.org/bash_cheat_sheet.pdf cheat sheets] and [http://www.tldp.org/LDP/Bash-Beginners-Guide/html/ tutorials] available on the web.

=== Typical Workflow ===
* Stage data to $USER_SCRATCH (your scratch directory)
* Research how to run your code in "batch" mode. Batch mode typically means the ability to run it from the command line without requiring any interaction from the user.
* Identify the appropriate resources needed to run the job. The following are mandatory resource requests for all jobs on Cheaha
** Number of processor cores required by the job
** Maximum memory (RAM) required per core
** Maximum runtime
* Write a job script specifying queuing system parameters, resource requests and commands to run program
* Submit script to queuing system (sbatch script.job)
* Monitor job (squeue)
* Review the results and resubmit as necessary
* Clean up the scratch directory by moving or deleting the data off of the cluster

=== Interactive Session ===
Head Node (The command-line interface after you login to Cheaha ) is supposed to be used for submitting jobs and/or lighter prep work required for the job scripts. '''You are not supposed to run heavy computations on the head node'''. If you have a heavier workload to prepare for a batch job (eg. compiling code or other manipulations of data) or your compute application requires interactive control, you should request a dedicated interactive node for this work.

Interactive resources are requested by submitting an "interactive" job to the scheduler. Interactive jobs will provide you a command line on a compute resource that you can use just like you would the command line on the head node. The difference is that the scheduler has dedicated the requested resources to your job and you can run your interactive commands without having to worry about impacting other users on the head node.
Interactive jobs are requested with the '''srun''' command

<pre>
srun -n 1 -N 1 -t 01:00:00 --pty /bin/bash
</pre>

This command requests for 1 core (-n) on 1 node (-N) for 1 hour (-t).

More advanced interactive scenarios to support graphical applications are available using [https://docs.uabgrid.uab.edu/wiki/Setting_Up_VNC_Session VNC] or X11 tunneling [http://www.uab.edu/it/software X-Win32 2014 for Windows]
== Updates ==

=== 20160311 Mike H ===

Howdy, the new changes are in place. The primary focus of the changes were to:
# Change the scheduling algorithm to one that allows jobs to share compute nodes (i.e. Slurm will allocate CPU cores now instead of complete compute nodes).
#We added partitions (in SGE they were called queues) with the following characteristics (these may change over time as we tweak things):
#* short (default partition): Priority 2 :: Max Runtime 2 hours
#* medium: Priority 4 :: Max Runtime 50 hours
#* long: Priority 6 :: Max Runtime 159 hours (6 days 6 hours)
#* interactive: Priority 10 :: Max Runtime 2 hours
# In order to run a job in a partition other than "short" you'll need to specifically request it using the --partition argument (--time=48:00:00 --partition=medium)
# Graphical interactive jobs now work. You can run an interactive job using the sinteractive command, for example:
sinteractive --time=00:05:00 --job-name=sinteractiveTest --ntasks=1 --mem=1024

More to come and please let us know of any issues or concerns.

Box.com bulk transfer

2016-02-25T16:28:23Z

Curtish@uab.edu: /* Warning: time stamps */

UAB has an Enterprise contract with Box.com, which is currently in BETA.

This page describes what we have learned about doing bulk-transfers of data.

== Warning: size limitations ==

Box.com claims to have a '''''5G max filesize''''' limit
* Now '''''15G max filesize''''' (2/18/016; private email), but "that file size limit is still considered to be in a beta phase"

There was a rumor this would be increased in early 2016.
Offically, this is all we know: https://community.box.com/t5/Managing-Your-Content/What-s-the-maximum-file-size-I-can-upload/ta-p/307

If you need to work around this, you can use the Linux "[http://ss64.com/bash/split.html split]" utility

{| class="wikitable"
|-
| # chop file into 4G pieces 
split \ 
--bytes=4000m \ 
big_file.fastq.gz \ 
big_file.fastq.gz.split4g. 
|-
| # record checksums of original and chunks 
md5sum \ 
big_file.fastq.gz \ 
big_file.fastq.gz.split4g.* \ 
> big_file.fastq.gz.md5
|}

== Warning: time stamps ==

When using an FTP client to transfer data up, it is easy to lose both modification and creation timestamps. In particular, many clients will (optionally) preserve modification time, but few will (optionally) preserve creation date.

{| class="wikitable" border="1"
|-
! FTP client
! cost
! platform
! preserve mod_date
! preserve create_date
|-
| lftp
| free
| linux/cmd_line
| yes
| no
|-
| SmartFTP
| $$
| Win Only/GUI
| yes
| can be enabled
|-
| fileZilla
| free
| linux & win/GUI_only
| can be enabled
| [https://trac.filezilla-project.org/ticket/2347 no]
|-
| ftp_ssl
| free
| linux/cmd_line
| yes
| no
|}

Filezilla on create times
* request closed, no plans to fix: https://trac.filezilla-project.org/ticket/2347

== Warning: Shared-to-you folders can't be moved ==

If someone creates a folder and shares it to you (as an reader, editor, co-owner, etc), it will live in your top level directory, and you will NOT be able to move it into any subfolder.

"Currently, users can't rearrange their own view of folders they are invited to collaborate within. As you note, when someone invites you to collaborate in a folder that you have never had access to before, you will see that folder on your root level." [box.com]

'''Workaround''': if your collaborator makes you the full owner of the folder, then you will be able to move it.

'''Fix timeframe''': "We've heard requests that people be able to rearrange their views before, and this is being considered as part of a larger product experience change next year" [box.com]

== lftp mirror -R examples (UPload) ==

lftp mirror
* "mirror" copies directory hierarchies DOWN from box.com to local
* "mirror -R" copies directory hierarchies UP from local to box.com

error handling
* the box server frequently looses connection (fails) on particular files
* just re-run the "mirror -R" and it will upload only new/failed files..

=== Interactive lftp ===
{| class="wikitable" border="1"
|-
| lftp ftp.box.com 
> user ''BLAZERID''@uab.edu ''users_BOX_external_password'' 
> mirror --parallel=10 -R ''local_src_dir'' ''box_dest_dir''
|}

=== Single-line lftp (non-shared box) ===

'''Warning''': This one makes the password visible to "ps" so should only be used on personal machines
{| class="wikitable" border="1"
|-
| lftp -u ''BLAZERID''@uab.edu,''users_BOX_external_password'' ftp.box.com << EOF 
mirror --parallel=10 -R ''local_src_dir'' ''box_dest_dir'' 
EOF
|}

=== scripted lftp ===
{| class="wikitable" border="1"
|-
| cat > box_upload.lftp << EOF 
open ftp.box.com 
user ''BLAZERID''@uab.edu ''users_BOX_external_password'' 
mirror --parallel=10 -R ''local_src_dir'' /''box_dest_dir'' 
EOF 
chmod 700 box_upload.lftp 
lftp -f box_upload.lftp ; echo lftp_RC=$?
|}

== lftp mirror examples (DOWNload) ==

=== scripted lftp ===

Arguments
* --loop keep restarting until there are no new files left to download - helps if someone else is uploading to that directory while you're downloading it!
* -v verbose level 1: includes bytes transferred and transfer speed.
* --parallel=10 use 10 concurrent TCP/IP connections (much faster)
{| class="wikitable" border="1"
|-
| cat > box_download.lftp << EOF 
open ftp.box.com 
user ''BLAZERID''@uab.edu ''users_BOX_external_password'' 
mirror --loop -v --parallel=10 /''box_remote_src_dir'' ''local_dest_dir'' 
EOF 
chmod 700 box_download.lftp 
lftp -f box_download.lftp ; echo lftp_RC=$?
|}

== Issues to resolve ==

* Routing over Iternet2
* we see our traffic randomly going over the commodity internet

== Linux Support ==
Unfortunately, Box doesn't provide a Linux client (is it on the road map?).

Box.com bulk transfer

2016-02-25T16:22:02Z

Curtish@uab.edu: /* Warning: size limitations */ raise to 15G

UAB has an Enterprise contract with Box.com, which is currently in BETA.

This page describes what we have learned about doing bulk-transfers of data.

== Warning: size limitations ==

Box.com claims to have a '''''5G max filesize''''' limit
* Now '''''15G max filesize''''' (2/18/016; private email), but "that file size limit is still considered to be in a beta phase"

There was a rumor this would be increased in early 2016.
Offically, this is all we know: https://community.box.com/t5/Managing-Your-Content/What-s-the-maximum-file-size-I-can-upload/ta-p/307

If you need to work around this, you can use the Linux "[http://ss64.com/bash/split.html split]" utility

{| class="wikitable"
|-
| # chop file into 4G pieces 
split \ 
--bytes=4000m \ 
big_file.fastq.gz \ 
big_file.fastq.gz.split4g. 
|-
| # record checksums of original and chunks 
md5sum \ 
big_file.fastq.gz \ 
big_file.fastq.gz.split4g.* \ 
> big_file.fastq.gz.md5
|}

== Warning: time stamps ==

When using an FTP client to transfer data up, it is easy to lose both modification and creation timestamps. In particular, many clients will (optionally) preserve modification time, but few will (optionally) preserve creation date.

{| class="wikitable" border="1"
|-
! FTP client
! platform
! modification
! creation
|-
| SmartFTP
| GUI/Win Only/$$
| yes
| can be enabled
|-
| lftp
| cmd_line/linux/free
| yes
| no
|-
| fileZilla
| GUI/linux+win/free
| can be enabled
| no
|-
| ftp_ssl
| cmd_line/linux/free
| yes
| no
|}

Filezilla on create times
* request closed, no plans to fix: https://trac.filezilla-project.org/ticket/2347

== Warning: Shared-to-you folders can't be moved ==

If someone creates a folder and shares it to you (as an reader, editor, co-owner, etc), it will live in your top level directory, and you will NOT be able to move it into any subfolder.

"Currently, users can't rearrange their own view of folders they are invited to collaborate within. As you note, when someone invites you to collaborate in a folder that you have never had access to before, you will see that folder on your root level." [box.com]

'''Workaround''': if your collaborator makes you the full owner of the folder, then you will be able to move it.

'''Fix timeframe''': "We've heard requests that people be able to rearrange their views before, and this is being considered as part of a larger product experience change next year" [box.com]

== lftp mirror -R examples (UPload) ==

lftp mirror
* "mirror" copies directory hierarchies DOWN from box.com to local
* "mirror -R" copies directory hierarchies UP from local to box.com

error handling
* the box server frequently looses connection (fails) on particular files
* just re-run the "mirror -R" and it will upload only new/failed files..

=== Interactive lftp ===
{| class="wikitable" border="1"
|-
| lftp ftp.box.com 
> user ''BLAZERID''@uab.edu ''users_BOX_external_password'' 
> mirror --parallel=10 -R ''local_src_dir'' ''box_dest_dir''
|}

=== Single-line lftp (non-shared box) ===

'''Warning''': This one makes the password visible to "ps" so should only be used on personal machines
{| class="wikitable" border="1"
|-
| lftp -u ''BLAZERID''@uab.edu,''users_BOX_external_password'' ftp.box.com << EOF 
mirror --parallel=10 -R ''local_src_dir'' ''box_dest_dir'' 
EOF
|}

=== scripted lftp ===
{| class="wikitable" border="1"
|-
| cat > box_upload.lftp << EOF 
open ftp.box.com 
user ''BLAZERID''@uab.edu ''users_BOX_external_password'' 
mirror --parallel=10 -R ''local_src_dir'' /''box_dest_dir'' 
EOF 
chmod 700 box_upload.lftp 
lftp -f box_upload.lftp ; echo lftp_RC=$?
|}

== lftp mirror examples (DOWNload) ==

=== scripted lftp ===

Arguments
* --loop keep restarting until there are no new files left to download - helps if someone else is uploading to that directory while you're downloading it!
* -v verbose level 1: includes bytes transferred and transfer speed.
* --parallel=10 use 10 concurrent TCP/IP connections (much faster)
{| class="wikitable" border="1"
|-
| cat > box_download.lftp << EOF 
open ftp.box.com 
user ''BLAZERID''@uab.edu ''users_BOX_external_password'' 
mirror --loop -v --parallel=10 /''box_remote_src_dir'' ''local_dest_dir'' 
EOF 
chmod 700 box_download.lftp 
lftp -f box_download.lftp ; echo lftp_RC=$?
|}

== Issues to resolve ==

* Routing over Iternet2
* we see our traffic randomly going over the commodity internet

== Linux Support ==
Unfortunately, Box doesn't provide a Linux client (is it on the road map?).

Box.com bulk transfer

2016-02-18T20:25:39Z

Curtish@uab.edu: /* scripted lftp */

UAB has an Enterprise contract with Box.com, which is currently in BETA.

This page describes what we have learned about doing bulk-transfers of data.

== Warning: size limitations ==

Box.com claims to have a '''''5G max filesize''''' limit.

There was a rumor this would be increased in early 2016.
Offically, this is all we know: https://community.box.com/t5/Managing-Your-Content/What-s-the-maximum-file-size-I-can-upload/ta-p/307

If you need to work around this, you can use the Linux "[http://ss64.com/bash/split.html split]" utility

{| class="wikitable"
|-
| # chop file into 4G pieces 
split \ 
--bytes=4000m \ 
big_file.fastq.gz \ 
big_file.fastq.gz.split4g. 
|-
| # record checksums of original and chunks 
md5sum \ 
big_file.fastq.gz \ 
big_file.fastq.gz.split4g.* \ 
> big_file.fastq.gz.md5
|}

== Warning: time stamps ==

When using an FTP client to transfer data up, it is easy to lose both modification and creation timestamps. In particular, many clients will (optionally) preserve modification time, but few will (optionally) preserve creation date.

{| class="wikitable" border="1"
|-
! FTP client
! platform
! modification
! creation
|-
| SmartFTP
| GUI/Win Only/$$
| yes
| can be enabled
|-
| lftp
| cmd_line/linux/free
| yes
| no
|-
| fileZilla
| GUI/linux+win/free
| can be enabled
| no
|-
| ftp_ssl
| cmd_line/linux/free
| yes
| no
|}

Filezilla on create times
* request closed, no plans to fix: https://trac.filezilla-project.org/ticket/2347

== Warning: Shared-to-you folders can't be moved ==

If someone creates a folder and shares it to you (as an reader, editor, co-owner, etc), it will live in your top level directory, and you will NOT be able to move it into any subfolder.

"Currently, users can't rearrange their own view of folders they are invited to collaborate within. As you note, when someone invites you to collaborate in a folder that you have never had access to before, you will see that folder on your root level." [box.com]

'''Workaround''': if your collaborator makes you the full owner of the folder, then you will be able to move it.

'''Fix timeframe''': "We've heard requests that people be able to rearrange their views before, and this is being considered as part of a larger product experience change next year" [box.com]

== lftp mirror -R examples (UPload) ==

lftp mirror
* "mirror" copies directory hierarchies DOWN from box.com to local
* "mirror -R" copies directory hierarchies UP from local to box.com

error handling
* the box server frequently looses connection (fails) on particular files
* just re-run the "mirror -R" and it will upload only new/failed files..

=== Interactive lftp ===
{| class="wikitable" border="1"
|-
| lftp ftp.box.com 
> user ''BLAZERID''@uab.edu ''users_BOX_external_password'' 
> mirror --parallel=10 -R ''local_src_dir'' ''box_dest_dir''
|}

=== Single-line lftp (non-shared box) ===

'''Warning''': This one makes the password visible to "ps" so should only be used on personal machines
{| class="wikitable" border="1"
|-
| lftp -u ''BLAZERID''@uab.edu,''users_BOX_external_password'' ftp.box.com << EOF 
mirror --parallel=10 -R ''local_src_dir'' ''box_dest_dir'' 
EOF
|}

=== scripted lftp ===
{| class="wikitable" border="1"
|-
| cat > box_upload.lftp << EOF 
open ftp.box.com 
user ''BLAZERID''@uab.edu ''users_BOX_external_password'' 
mirror --parallel=10 -R ''local_src_dir'' /''box_dest_dir'' 
EOF 
chmod 700 box_upload.lftp 
lftp -f box_upload.lftp ; echo lftp_RC=$?
|}

== lftp mirror examples (DOWNload) ==

=== scripted lftp ===

Arguments
* --loop keep restarting until there are no new files left to download - helps if someone else is uploading to that directory while you're downloading it!
* -v verbose level 1: includes bytes transferred and transfer speed.
* --parallel=10 use 10 concurrent TCP/IP connections (much faster)
{| class="wikitable" border="1"
|-
| cat > box_download.lftp << EOF 
open ftp.box.com 
user ''BLAZERID''@uab.edu ''users_BOX_external_password'' 
mirror --loop -v --parallel=10 /''box_remote_src_dir'' ''local_dest_dir'' 
EOF 
chmod 700 box_download.lftp 
lftp -f box_download.lftp ; echo lftp_RC=$?
|}

== Issues to resolve ==

* Routing over Iternet2
* we see our traffic randomly going over the commodity internet

Slurm

2016-01-20T18:29:46Z

Curtish@uab.edu: /* Interactive Session */

Box.com bulk transfer

2016-01-15T16:47:12Z

Curtish@uab.edu: /* lftp mirror -R example */

UAB has an Enterprise contract with Box.com, which is currently in BETA.

This page describes what we have learned about doing bulk-transfers of data.

== Warning: size limitations ==

Box.com claims to have a '''''5G max filesize''''' limit.

There was a rumor this would be increased in early 2016.
Offically, this is all we know: https://community.box.com/t5/Managing-Your-Content/What-s-the-maximum-file-size-I-can-upload/ta-p/307

If you need to work around this, you can use the Linux "[http://ss64.com/bash/split.html split]" utility

{| class="wikitable"
|-
| # chop file into 4G pieces 
split \ 
--bytes=4000m \ 
big_file.fastq.gz \ 
big_file.fastq.gz.split4g. 
|-
| # record checksums of original and chunks 
md5sum \ 
big_file.fastq.gz \ 
big_file.fastq.gz.split4g.* \ 
> big_file.fastq.gz.md5
|}

== Warning: time stamps ==

When using an FTP client to transfer data up, it is easy to lose both modification and creation timestamps. In particular, many clients will (optionally) preserve modification time, but few will (optionally) preserve creation date.

{| class="wikitable" border="1"
|-
! FTP client
! platform
! modification
! creation
|-
| SmartFTP
| GUI/Win Only/$$
| yes
| can be enabled
|-
| lftp
| cmd_line/linux/free
| yes
| no
|-
| fileZilla
| GUI/linux+win/free
| can be enabled
| no
|-
| ftp_ssl
| cmd_line/linux/free
| yes
| no
|}

Filezilla on create times
* request closed, no plans to fix: https://trac.filezilla-project.org/ticket/2347

== Warning: Shared-to-you folders can't be moved ==

If someone creates a folder and shares it to you (as an reader, editor, co-owner, etc), it will live in your top level directory, and you will NOT be able to move it into any subfolder.

"Currently, users can't rearrange their own view of folders they are invited to collaborate within. As you note, when someone invites you to collaborate in a folder that you have never had access to before, you will see that folder on your root level." [box.com]

'''Workaround''': if your collaborator makes you the full owner of the folder, then you will be able to move it.

'''Fix timeframe''': "We've heard requests that people be able to rearrange their views before, and this is being considered as part of a larger product experience change next year" [box.com]

== lftp mirror -R examples (UPload) ==

lftp mirror
* "mirror" copies directory hierarchies DOWN from box.com to local
* "mirror -R" copies directory hierarchies UP from local to box.com

error handling
* the box server frequently looses connection (fails) on particular files
* just re-run the "mirror -R" and it will upload only new/failed files..

=== Interactive lftp ===
{| class="wikitable" border="1"
|-
| lftp ftp.box.com 
> user ''BLAZERID''@uab.edu ''users_BOX_external_password'' 
> mirror --parallel=10 -R ''local_src_dir'' ''box_dest_dir''
|}

=== Single-line lftp (non-shared box) ===

'''Warning''': This one makes the password visible to "ps" so should only be used on personal machines
{| class="wikitable" border="1"
|-
| lftp -u ''BLAZERID''@uab.edu,''users_BOX_external_password'' ftp.box.com << EOF 
mirror --parallel=10 -R ''local_src_dir'' ''box_dest_dir'' 
EOF
|}

=== scripted lftp ===
{| class="wikitable" border="1"
|-
| cat > box_upload.lftp << EOF 
open ftp.box.com 
user ''BLAZERID''@uab.edu ''users_BOX_external_password'' 
mirror --parallel=10 -R ''local_src_dir'' /''box_dest_dir'' 
EOF 
chmod 700 box_upload.lftp 
lftp -f box_upload.lftp ; echo lftp_RC=$?
|}

== lftp mirror examples (DOWNload) ==

=== scripted lftp ===

Arguments
* --loop keep restarting until there are no new files left to download - helps if someone else is uploading to that directory while you're downloading it!
* -v verbose level 1: includes bytes transferred and transfer speed.
* --parallel=10 use 10 concurrent TCP/IP connections (much faster)
{| class="wikitable" border="1"
|-
| cat > box_download.lftp << EOF 
open ftp.box.com 
user ''BLAZERID''@uab.edu ''users_BOX_external_password'' 
mirror --loop -v --parallel=10 /''box_remote_src_dir'' ''local_dest_dir'' 
EOF 
chmod 700 box_download.lftp 
lftp -f box_download.lftp ; echo lftp_RC=$?
|}

Box.com bulk transfer

2015-12-16T21:29:08Z

Curtish@uab.edu: /* lftp mirror -R example */

UAB has an Enterprise contract with Box.com, which is currently in BETA.

This page describes what we have learned about doing bulk-transfers of data.

== Warning: size limitations ==

Box.com claims to have a '''''5G max filesize''''' limit.

There was a rumor this would be increased in early 2016.
Offically, this is all we know: https://community.box.com/t5/Managing-Your-Content/What-s-the-maximum-file-size-I-can-upload/ta-p/307

If you need to work around this, you can use the Linux "[http://ss64.com/bash/split.html split]" utility

{| class="wikitable"
|-
| # chop file into 4G pieces 
split \ 
--bytes=4000m \ 
big_file.fastq.gz \ 
big_file.fastq.gz.split4g. 
|-
| # record checksums of original and chunks 
md5sum \ 
big_file.fastq.gz \ 
big_file.fastq.gz.split4g.* \ 
> big_file.fastq.gz.md5
|}

== Warning: time stamps ==

When using an FTP client to transfer data up, it is easy to lose both modification and creation timestamps. In particular, many clients will (optionally) preserve modification time, but few will (optionally) preserve creation date.

{| class="wikitable" border="1"
|-
! FTP client
! platform
! modification
! creation
|-
| SmartFTP
| GUI/Win Only/$$
| yes
| can be enabled
|-
| lftp
| cmd_line/linux/free
| yes
| no
|-
| fileZilla
| GUI/linux+win/free
| can be enabled
| no
|-
| ftp_ssl
| cmd_line/linux/free
| yes
| no
|}

Filezilla on create times
* request closed, no plans to fix: https://trac.filezilla-project.org/ticket/2347

== Warning: Shared-to-you folders can't be moved ==

If someone creates a folder and shares it to you (as an reader, editor, co-owner, etc), it will live in your top level directory, and you will NOT be able to move it into any subfolder.

"Currently, users can't rearrange their own view of folders they are invited to collaborate within. As you note, when someone invites you to collaborate in a folder that you have never had access to before, you will see that folder on your root level." [box.com]

'''Workaround''': if your collaborator makes you the full owner of the folder, then you will be able to move it.

'''Fix timeframe''': "We've heard requests that people be able to rearrange their views before, and this is being considered as part of a larger product experience change next year" [box.com]

== lftp mirror -R example ==

lftp mirror
* "mirror" copies directory hierarchies DOWN from box.com to local
* "mirror -R" copies directory hierarchies UP from local to box.com

error handling
* the box server frequently looses connection (fails) on particular files
* just re-run the "mirror -R" and it will upload only new/failed files..

=== Interactive lftp ===
{| class="wikitable" border="1"
|-
| lftp ftp.box.com 
> user ''BLAZERID''@uab.edu ''users_BOX_external_password'' 
> mirror --parallel=10 -R ''local_src_dir'' ''box_dest_dir''
|}

=== Single-line lftp (non-shared box) ===

'''Warning''': This one makes the password visible to "ps" so should only be used on personal machines
{| class="wikitable" border="1"
|-
| lftp -u ''BLAZERID''@uab.edu,''users_BOX_external_password'' ftp.box.com << EOF 
mirror --parallel=10 -R ''local_src_dir'' ''box_dest_dir'' 
EOF
|}

=== scripted lftp ===
{| class="wikitable" border="1"
|-
| cat > box_test.lftp << EOF 
open ftp.box.com 
user ''BLAZERID''@uab.edu ''users_BOX_external_password'' 
mirror --parallel=10 -R ''local_src_dir'' /''box_dest_dir'' 
EOF 
chmod 700 box_test.lftp 
lftp -f box_test.lftp ; echo lftp_RC=$?
|}

Box.com bulk transfer

2015-12-16T18:28:38Z

Curtish@uab.edu:

UAB has an Enterprise contract with Box.com, which is currently in BETA.

This page describes what we have learned about doing bulk-transfers of data.

== Warning: size limitations ==

Box.com claims to have a '''''5G max filesize''''' limit.

There was a rumor this would be increased in early 2016.
Offically, this is all we know: https://community.box.com/t5/Managing-Your-Content/What-s-the-maximum-file-size-I-can-upload/ta-p/307

If you need to work around this, you can use the Linux "[http://ss64.com/bash/split.html split]" utility

{| class="wikitable"
|-
| # chop file into 4G pieces 
split \ 
--bytes=4000m \ 
big_file.fastq.gz \ 
big_file.fastq.gz.split4g. 
|-
| # record checksums of original and chunks 
md5sum \ 
big_file.fastq.gz \ 
big_file.fastq.gz.split4g.* \ 
> big_file.fastq.gz.md5
|}

== Warning: time stamps ==

When using an FTP client to transfer data up, it is easy to lose both modification and creation timestamps. In particular, many clients will (optionally) preserve modification time, but few will (optionally) preserve creation date.

{| class="wikitable" border="1"
|-
! FTP client
! platform
! modification
! creation
|-
| SmartFTP
| GUI/Win Only/$$
| yes
| can be enabled
|-
| lftp
| cmd_line/linux/free
| yes
| no
|-
| fileZilla
| GUI/linux+win/free
| can be enabled
| no
|-
| ftp_ssl
| cmd_line/linux/free
| yes
| no
|}

Filezilla on create times
* request closed, no plans to fix: https://trac.filezilla-project.org/ticket/2347

== Warning: Shared-to-you folders can't be moved ==

If someone creates a folder and shares it to you (as an reader, editor, co-owner, etc), it will live in your top level directory, and you will NOT be able to move it into any subfolder.

"Currently, users can't rearrange their own view of folders they are invited to collaborate within. As you note, when someone invites you to collaborate in a folder that you have never had access to before, you will see that folder on your root level." [box.com]

'''Workaround''': if your collaborator makes you the full owner of the folder, then you will be able to move it.

'''Fix timeframe''': "We've heard requests that people be able to rearrange their views before, and this is being considered as part of a larger product experience change next year" [box.com]

== lftp mirror -R example ==

lftp mirror
* "mirror" copies directory hierarchies DOWN from box.com to local
* "mirror -R" copies directory hierarchies UP from local to box.com

error handling
* the box server frequently looses connection (fails) on particular files
* just re-run the "mirror -R" and it will upload only new/failed files..

=== Interactive lftp ===
{| class="wikitable" border="1"
|-
| lftp ftp.box.com 
> user ''BLAZERID''@uab.edu ''users_BOX_external_password'' 
> mirror --parallel=10 -R ''local_src_dir'' ''box_dest_dir''
|}

=== scripted lftp ===
{| class="wikitable" border="1"
|-
| cat > box_test.lftp << EOF 
open ftp.box.com 
user ''BLAZERID''@uab.edu ''users_BOX_external_password'' 
mirror --parallel=10 -R ''local_src_dir'' /''box_dest_dir'' 
EOF 
chmod 700 box_test.lftp 
lftp -f box_test.lftp ; echo lftp_RC=$?
|}

Box.com bulk transfer

2015-12-16T17:30:51Z

Curtish@uab.edu:

UAB has an Enterprise contract with Box.com, which is currently in BETA.

This page describes what we have learned about doing bulk-transfers of data.

== Warning: size limitations ==

Box.com claims to have a '''''5G max filesize''''' limit.

There was a rumor this would be increased in early 2016.
Offically, this is all we know: https://community.box.com/t5/Managing-Your-Content/What-s-the-maximum-file-size-I-can-upload/ta-p/307

If you need to work around this, you can use the Linux "[http://ss64.com/bash/split.html split]" utility

{| class="wikitable"
|-
| # chop file into 4G pieces 
split \ 
--bytes=4000m \ 
big_file.fastq.gz \ 
big_file.fastq.gz.split4g. 
|-
| # record checksums of original and chunks 
md5sum \ 
big_file.fastq.gz \ 
big_file.fastq.gz.split4g.* \ 
> big_file.fastq.gz.md5
|}

== Warning: time stamps ==

When using an FTP client to transfer data up, it is easy to lose both modification and creation timestamps. In particular, many clients will (optionally) preserve modification time, but few will (optionally) preserve creation date.

{| class="wikitable" border="1"
|-
! FTP client
! platform
! modification
! creation
|-
| SmartFTP
| GUI/Win Only/$$
| yes
| can be enabled
|-
| lftp
| cmd_line/linux/free
| yes
| no
|-
| fileZilla
| GUI/linux+win/free
| can be enabled
| no
|-
| ftp_ssl
| cmd_line/linux/free
| yes
| no
|}

Filezilla on create times
* request closed, no plans to fix: https://trac.filezilla-project.org/ticket/2347

== lftp mirror -R example ==

lftp mirror
* "mirror" copies directory hierarchies DOWN from box.com to local
* "mirror -R" copies directory hierarchies UP from local to box.com

error handling
* the box server frequently looses connection (fails) on particular files
* just re-run the "mirror -R" and it will upload only new/failed files..

=== Interactive lftp ===
{| class="wikitable" border="1"
|-
| lftp ftp.box.com 
> user ''BLAZERID''@uab.edu ''users_BOX_external_password'' 
> mirror --parallel=10 -R ''local_src_dir'' ''box_dest_dir''
|}

=== scripted lftp ===
{| class="wikitable" border="1"
|-
| cat > box_test.lftp << EOF 
open ftp.box.com 
user ''BLAZERID''@uab.edu ''users_BOX_external_password'' 
mirror --parallel=10 -R ''local_src_dir'' /''box_dest_dir'' 
EOF 
chmod 700 box_test.lftp 
lftp -f box_test.lftp ; echo lftp_RC=$?
|}

Box.com bulk transfer

2015-12-15T22:10:13Z

Curtish@uab.edu: /* Warning: time stamps */

UAB has an Enterprise contract with Box.com, which is currently in BETA.

This page describes what we have learned about doing bulk-transfers of data.

== Warning: size limitations ==

Box.com claims to have a '''''5G max filesize''''' limit.

There was a rumor this would be increased in early 2016.
Offically, this is all we know: https://community.box.com/t5/Managing-Your-Content/What-s-the-maximum-file-size-I-can-upload/ta-p/307

If you need to work around this, you can use the Linux "[http://ss64.com/bash/split.html split]" utility

{| class="wikitable"
|-
| # chop file into 4G pieces 
split \ 
--bytes=4000m \ 
big_file.fastq.gz \ 
big_file.fastq.gz.split4g. 
|-
| # record checksums of original and chunks 
md5sum \ 
big_file.fastq.gz \ 
big_file.fastq.gz.split4g.* \ 
> big_file.fastq.gz.md5
|}

== Warning: time stamps ==

When using an FTP client to transfer data up, it is easy to lose both modification and creation timestamps. In particular, many clients will (optionally) preserve modification time, but few will (optionally) preserve creation date.

{| class="wikitable" border="1"
|-
! FTP client
! platform
! modification
! creation
|-
| SmartFTP
| GUI/Win Only/$$
| yes
| can be enabled
|-
| lftp
| cmd_line/linux/free
| yes
| no
|-
| fileZilla
| GUI/linux+win/free
| can be enabled
| no
|-
| ftp_ssl
| cmd_line/linux/free
| yes
| no
|}

Filezilla on create times
request closed, no plans to fix: https://trac.filezilla-project.org/ticket/2347

* Bulleted list item
* Bulleted list item

Box.com bulk transfer

2015-12-15T20:18:12Z

Curtish@uab.edu: /* Warning: size limitations */

UAB has an Enterprise contract with Box.com, which is currently in BETA.

This page describes what we have learned about doing bulk-transfers of data.

== Warning: size limitations ==

Box.com claims to have a '''''5G max filesize''''' limit.

There was a rumor this would be increased in early 2016.
Offically, this is all we know: https://community.box.com/t5/Managing-Your-Content/What-s-the-maximum-file-size-I-can-upload/ta-p/307

If you need to work around this, you can use the Linux "[http://ss64.com/bash/split.html split]" utility

{| class="wikitable"
|-
| # chop file into 4G pieces 
split \ 
--bytes=4000m \ 
big_file.fastq.gz \ 
big_file.fastq.gz.split4g. 
|-
| # record checksums of original and chunks 
md5sum \ 
big_file.fastq.gz \ 
big_file.fastq.gz.split4g.* \ 
> big_file.fastq.gz.md5
|}

== Warning: time stamps ==

When using an FTP client to transfer data up, it is easy to lose both modification and creation timestamps. In particular, many clients will (optionally) preserve modification time, but few will (optionally) preserve creation date.

{| class="wikitable" border="1"
|-
! FTP client
! platform
! modification
! creation
|-
| SmartFTP
| GUI/Win Only/$$
| yes
| can be enabled
|-
| lftp
| cmd_line/linux/free
| yes
| no
|-
| fileZilla
| GUI/linux+win/free
| can be enabled
| no
|-
| ftp_ssl
| cmd_line/linux/free
| yes
| no
|}

Box.com bulk transfer

2015-12-15T18:52:24Z

Curtish@uab.edu: /* Warning: size limitations */

UAB has an Enterprise contract with Box.com, which is currently in BETA.

This page describes what we have learned about doing bulk-transfers of data.

== Warning: size limitations ==

Box.com claims to have a 5G max filesize limit, which will be removed in 2016.
If you need to work around this, you can use the Linux "[http://ss64.com/bash/split.html split]" utility

{| class="wikitable"
|-
| # chop file into 4G pieces 
split \ 
--bytes=4000m \ 
big_file.fastq.gz \ 
big_file.fastq.gz.split4g. 
|-
| # record checksums of original and chunks 
md5sum \ 
big_file.fastq.gz \ 
big_file.fastq.gz.split4g.* \ 
> big_file.fastq.gz.md5
|}

== Warning: time stamps ==

When using an FTP client to transfer data up, it is easy to lose both modification and creation timestamps. In particular, many clients will (optionally) preserve modification time, but few will (optionally) preserve creation date.

{| class="wikitable" border="1"
|-
! FTP client
! platform
! modification
! creation
|-
| SmartFTP
| GUI/Win Only/$$
| yes
| can be enabled
|-
| lftp
| cmd_line/linux/free
| yes
| no
|-
| fileZilla
| GUI/linux+win/free
| can be enabled
| no
|-
| ftp_ssl
| cmd_line/linux/free
| yes
| no
|}

Box.com bulk transfer

2015-12-15T18:47:48Z

Curtish@uab.edu: /* Warning: size limitations */

UAB has an Enterprise contract with Box.com, which is currently in BETA.

This page describes what we have learned about doing bulk-transfers of data.

== Warning: size limitations ==

Box.com claims to have a 5G max filesize limit, which will be removed in 2016.
If you need to work around this, you can use the Linux "[http://ss64.com/bash/split.html split]" utility

{| class="wikitable"
|-
| split \ 
--bytes=4000m \ 
big_file.fastq.gz \ 
big_file.fastq.gz.split4g.
|}

== Warning: time stamps ==

When using an FTP client to transfer data up, it is easy to lose both modification and creation timestamps. In particular, many clients will (optionally) preserve modification time, but few will (optionally) preserve creation date.

{| class="wikitable" border="1"
|-
! FTP client
! platform
! modification
! creation
|-
| SmartFTP
| GUI/Win Only/$$
| yes
| can be enabled
|-
| lftp
| cmd_line/linux/free
| yes
| no
|-
| fileZilla
| GUI/linux+win/free
| can be enabled
| no
|-
| ftp_ssl
| cmd_line/linux/free
| yes
| no
|}

Box.com bulk transfer

2015-12-07T20:49:21Z

Curtish@uab.edu: start list of FTP programs to push data to box.com

UAB has an Enterprise contract with Box.com, which is currently in BETA.

This page describes what we have learned about doing bulk-transfers of data.

== Warning: size limitations ==

Box.com claims to have a 5G max filesize limit, which will be removed in 2016.
If you need to work around this, you can use the Linux "[http://ss64.com/bash/split.html split]" utility

<nowiki>:Indented line
split --bytes=4000m big_file.fastq.gz big_file.fastq.gz.split4g.
</nowiki>

== Warning: time stamps ==

When using an FTP client to transfer data up, it is easy to lose both modification and creation timestamps. In particular, many clients will (optionally) preserve modification time, but few will (optionally) preserve creation date.

{| class="wikitable" border="1"
|-
! FTP client
! platform
! modification
! creation
|-
| SmartFTP
| GUI/Win Only/$$
| yes
| can be enabled
|-
| lftp
| cmd_line/linux/free
| yes
| no
|-
| fileZilla
| GUI/linux+win/free
| can be enabled
| no
|-
| ftp_ssl
| cmd_line/linux/free
| yes
| no
|}