Cheaha - User contributions [en]

ScreenCommand

2013-09-26T16:38:53Z

Pavgi@uab.edu: added screen cli options

As the [http://www.gnu.org/software/screen/manual/screen.html man page] states, Screen is a full-screen window manager that multiplexes a physical terminal between several processes (typically interactive shells). A process (shell or command like ls) can be attached to the screen command/session. The screen session won't die or exit until terminated by the attached command/shell. This is useful when working over a flaky network connection. Screen can multiplex commands into a single window so it makes it convenient to open new shell sessions in the same window.

== Starting a screen session ==
You can start a new screen session just by typing 'screen' command. You can name your screen session using -S option as follows:
<pre>
screen -S screen-getting-started
</pre>

== Screen control keystrokes ==
{| class="wikitable"
|-
! Action !! Keystroke
|-
| Help menu || Ctrl-a ?
|-
| Version || Ctrl-a v
|-
| List windows || Ctrl-a w
|-
| Scroll through windows || Ctrl-a Ctrl-a
|-
| Rename window session || Ctrl-a A
|-
| Detach from the screen session || Ctrl-a d
|}

== Screen command options ==
{| class="wikitable"
|-
! Action !! Option
|-
| List screen sessions || screen -list or screen -ls
|-
| Reattach to a screen session || screen -r <session-name>
|}

== Adding configuration options to screenrc file ==
You can customize screen settings using command-line options or by creating a screenrc file. See [http://www.gnu.org/software/screen/manual/screen.html#Customization Customization] section in the man page for detailed options. See [http://magazine.redhat.com/2007/09/27/a-guide-to-gnu-screen/ Red Hat magazine article] for examples.

== Logging screen window to a screenlog file ==
Screen can log it's window output to a screenlog file. This can be done by invoking screen with -L option.

== Viewing screenlog file ==
Screen log file contains escape characters and hence it's not directly human readable. You can get pretty output using less command with -R or -r options. See [http://superuser.com/questions/99128/removing-the-escape-characters-from-gnu-screens-screenlog-n stackoverflow answers] for hints.

ScreenCommand

2013-09-26T16:30:10Z

Pavgi@uab.edu: added quick start command reference

As the [http://www.gnu.org/software/screen/manual/screen.html man page] states, Screen is a full-screen window manager that multiplexes a physical terminal between several processes (typically interactive shells). A process (shell or command like ls) can be attached to the screen command/session. The screen session won't die or exit until terminated by the attached command/shell. This is useful when working over a flaky network connection. Screen can multiplex commands into a single window so it makes it convenient to open new shell sessions in the same window.

== Starting a screen session ==
You can start a new screen session just by typing 'screen' command. You can name your screen session using -S option as follows:
<pre>
screen -S screen-getting-started
</pre>

== Quick start ==
{| class="wikitable"
|-
! Action !! Keystroke
|-
| Help menu || Ctrl-a ?
|-
| Version || Ctrl-a v
|-
| List windows || Ctrl-a w
|-
| Scroll through windows || Ctrl-a Ctrl-a
|-
| Rename window session || Ctrl-a A
|}

== Adding configuration options to screenrc file ==
You can customize screen settings using command-line options or by creating a screenrc file. See [http://www.gnu.org/software/screen/manual/screen.html#Customization Customization] section in the man page for detailed options. See [http://magazine.redhat.com/2007/09/27/a-guide-to-gnu-screen/ Red Hat magazine article] for examples.

== Logging screen window to a screenlog file ==
Screen can log it's window output to a screenlog file. This can be done by invoking screen with -L option.

== Viewing screenlog file ==
Screen log file contains escape characters and hence it's not directly human readable. You can get pretty output using less command with -R or -r options. See [http://superuser.com/questions/99128/removing-the-escape-characters-from-gnu-screens-screenlog-n stackoverflow answers] for hints.

ScreenCommand

2013-07-18T19:40:29Z

Pavgi@uab.edu:

As the [http://www.gnu.org/software/screen/manual/screen.html man page] states, Screen is a full-screen window manager that multiplexes a physical terminal between several processes (typically interactive shells). A process (shell or command like ls) can be attached to the screen command/session. The screen session won't die or exit until terminated by the attached command/shell. This is useful when working over a flaky network connection. Screen can multiplex commands into a single window so it makes it convenient to open new shell sessions in the same window.

== Starting a screen session ==
You can start a new screen session just by typing 'screen' command. You can name your screen session using -S option as follows:
<pre>
screen -S screen-getting-started
</pre>

== Adding configuration options to screenrc file ==
You can customize screen settings using command-line options or by creating a screenrc file. See [http://www.gnu.org/software/screen/manual/screen.html#Customization Customization] section in the man page for detailed options. See [http://magazine.redhat.com/2007/09/27/a-guide-to-gnu-screen/ Red Hat magazine article] for examples.

== Logging screen window to a screenlog file ==
Screen can log it's window output to a screenlog file. This can be done by invoking screen with -L option.

== Viewing screenlog file ==
Screen log file contains escape characters and hence it's not directly human readable. You can get pretty output using less command with -R or -r options. See [http://superuser.com/questions/99128/removing-the-escape-characters-from-gnu-screens-screenlog-n stackoverflow answers] for hints.

ScreenCommand

2013-07-18T19:37:05Z

Pavgi@uab.edu: added screen command intro

As the [http://www.gnu.org/software/screen/manual/screen.html man page] states, Screen is a full-screen window manager that multiplexes a physical terminal between several processes (typically interactive shells). A process (shell or command like ls) can be attached to the screen command/session. The screen session won't die or exit until terminated by the attached command/shell. This is useful when SSH connections are opened over flaky network connections. Screen can multiplex commands into a single window so it makes it convenient to open new shell sessions in the same window.

== Starting a screen session ==
You can start a new screen session just by typing 'screen' command. You can name your screen session using -S option as follows:
<pre>
screen -S screen-getting-started
</pre>

== Adding configuration options to screenrc file ==
You can customize screen settings using command-line options or by creating a screenrc file. See [http://www.gnu.org/software/screen/manual/screen.html#Customization Customization] section in the man page for detailed options. See [http://magazine.redhat.com/2007/09/27/a-guide-to-gnu-screen/ Red Hat magazine article] for examples.

== Logging screen window to a screenlog file ==
Screen can log it's window output to a screenlog file. This can be done by invoking screen with -L option.

== Viewing screenlog file ==
Screen log file contains escape characters and hence it's not directly human readable. You can get pretty output using less command with -R or -r options. See [http://superuser.com/questions/99128/removing-the-escape-characters-from-gnu-screens-screenlog-n stackoverflow answers] for hints.

TopHat

2013-06-18T19:06:38Z

Pavgi@uab.edu: /* TopHat Tutorials */

[http://tophat.cbcb.umd.edu/ TopHat] is a fast splice junction mapper for RNA-Seq reads. It aligns RNA-Seq reads to mammalian-sized genomes using the ultra high-throughput short read aligner [http://bowtie-bio.sourceforge.net/index.shtml Bowtie], and then analyzes the mapping results to identify splice junctions between exons.

TopHat is a collaborative effort between the [http://cbcb.umd.edu/ University of Maryland Center for Bioinformatics and Computational Biology] and the University of California, Berkeley [http://math.berkeley.edu/ Departments of Mathematics] and [http://www.mcb.berkeley.edu/ Molecular and Cell Biology].

The latest stable release on [[Cheaha]] is 1.2.0 as part of the [http://main.g2.bx.psu.edu/ Galaxy] module. TopHat can run through the [http://galaxy.uabgrid.uab.edu/ UAB Galaxy] interface or directly on the cluster by submitting jobs to SGE.

==Using TopHat==
TopHat is a free software that runs on Linux and Mac OS X.

===TopHat on your Desktop ===
TopHat can be downloaded and installed on your desktop from http://tophat.cbcb.umd.edu/

Installation instructions for TopHat are available at: http://tophat.cbcb.umd.edu/tutorial.html

===TopHat on Cheaha===
TopHat is pre-installed on the [[Cheaha]] research computing system. This allows users to run TopHat directly on the cluster without any need to install software.

There are two methods of using TopHat on Cheaha:
1. Direct use on the cluster using SGE submit scripts.
2. TopHat use through the [http://galaxy.uabgrid.uab.edu/ UAB Galaxy] interface.

====Direct Use through SGE submit scripts ====

These instructions provide an example of how to create and submit a TopHat job on [[Cheaha]].

First, create the working directory for the job- Replace 'USERNAME' with the account associated username on Cheaha.
You can create any directory to run your job. It is recommended that the job directory be on the scratch (i.e. lustre filesystem) instead of the user home directory.

<pre>
$ mkdir -p /lustre/scratch/USERNAME/jobs/tophat
$ cd /lustre/scratch/USERNAME/jobs/tophat
</pre>

Next, Copy all the files required for TopHat to the working directory.

TopHat requires at least the reference index database and the sequence reads files.
If the Genome index database is not already built, you build it using the following steps.
Load the appropriate modules file to load TopHat and Bowtie.

<pre>
$ module load module load galaxy/galaxy
$ bowtie-build /lustre/scratch/USERNAME/jobs/tophat/example.fa example_bowtie_idx
</pre>
This builds the Bowtie index for the genome 'example'. You can also download the pre-built index file from other external sources.

Next, create a job submit script as shown below called 'tophatSubmit', make sure to edit the following parameters:

* s_rt to an appropriate soft wall time limit
* h_rt to the maximum wall time for your job
* -N - job name
* -M - user email
* -pe smp numberOfProcessors (-pe smp 8 - run the code in parallel on 8 processors on the same node of Cheaha.
Also, -pe smp 12 is the maximum available number of processors)
* -l vf to the maximum memory needed for each task
* In the following example the genome is 'example', the reads are read1,read2, output will be written to testTophat/ folder.
The -r parameter is the Mean Inner Distance between Mate Pairs for pair-ended library.
This parameter can be left out for a single-ended library.

<pre>
#!/bin/bash
#$ -S /bin/bash
#
# Execute in the current working directory
#$ -cwd
#
# Job runtime (10 hours)
#$ -l h_rt=10:00:00,s_rt=9:55:00
#$ -j y
#
# Job Name and email
#$ -N tophat
#$ -M username@uab.edu
#
#$ -pe smp 8
# Load the appropriate module(s)
module load galaxy/galaxy
#
#$ -V
# Amount of Memory needed (RAM) 3G
#$ -l vf=3G

tophat -p 8 -r 20 -o testTophat/ example read1 read2

</pre>

Submit the script to the scheduler with
<pre>
$ qsub tophatSubmit
</pre>

The output will be
<pre>
Your job 8078636 ("tophat") has been submitted
</pre>

You can check the status of the jobs using the 'qstat' command
<pre>
qstat -u $USER

job-ID prior name user state submit/start at queue slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
8078636 0.52111 tophat tanthony r 08/03/2011 11:42:41 sipsey.q@sipsey-compute-2-15.l 8
</pre>

The job output can be found in the output directory specified in the submit script.
You can view the output using samtools.
<pre>
$ cd testTophat
$ samtools view accepted_hits.bam
</pre>

====Using the UAB Galaxy interface====
# Login to the UAB Galaxy interface at: http://galaxy.uabgrid.uab.edu/
# Upload the sequences file(s) using the 'Get Data' tool. (data should be available in the History pane)
# Click 'NGS:RNA Analyasis' > 'Tophat' under the NGS Toolbox
# Select reference genome and RNA sequence file
# Execute

== Galaxy support ==
In order to facilitate interaction among UAB Galaxy users, share experience, and provide peer-support we have established a galaxy-users group. To join this group and participate in email discussions please subscribe to the [https://vo.uabgrid.uab.edu/sympa/subscribe/galaxy-user galaxy-user] group. On-line archives of these discussions are available [https://vo.uabgrid.uab.edu/sympa/arc/galaxy-user here]. Please note, the email discussions are a public forum. You are advised to only post information you are authorized to share and comfortable with being public.

== TopHat Tutorials ==
* TopHat Tutorials can be found at: http://tophat.cbcb.umd.edu/manual.html
* TopHat Tutorials: http://icb.med.cornell.edu/wiki/index.php/Tophat_tutorial

[[Category: NGS]][[Category:Software]][[Category:Bioinformatics]]

TopHat

2013-06-18T19:02:50Z

Pavgi@uab.edu: /* TopHat Tutorials */

Cheaha Quick Start Interactive Jobs

2013-04-09T20:47:20Z

Pavgi@uab.edu:

'''NOTE: This page is still under development. Please refer to [[Cheaha_GettingStarted]] page for detailed documentation.'''

= Interactive Resources =

After you login to Cheaha the command-line interface that you see is running on the head node. Most of your light interactive prep-work to submit a compute job to the scheduler can be carried out on this head node. If you have a heavier workload to prepare for a batch job (eg. compiling code or other manipulations of data) or your compute application requires interactive control, you should request a dedicated interactive node for this work.

Interactive resources are requested by submitting an "interactive" job to the scheduler. Interactive jobs will provide you a command line on a compute resource that you can use just like you would the command line on the head node. The difference is that the scheduler has dedicated the requested resources to your job and you can run your interactive commands without having to worry about impacting other users on the head node.

Interactive jobs are requested with the qrsh or qlogin command (please use your correct email address in place of the ''$USER@uab.edu'' string if you do not have an @uab.edu email address):
<pre>
qlogin -l h_rt=1:00:00,vf=1G -m b -M $USER@uab.edu -now no
</pre>

This command requests 1 compute core with 1Gb of RAM for 1 hour. The command will wait until the resource is reserved by the scheduler and send you an email when the resource is available. The email alert can be useful during periods of heavy cluster demand when interactive resources reservations may have significant wait times.

= Examples =
Below are some examples on launching interactive jobs on Cheaha.

== Code compilation or command line work using single core ==
Following example uses default SSH options to login to Cheaha, launch interactive job to get a compute node and compile Python. This approach can be used to run any code compilation or perform any command line activity. If your actions are going to involve more than 1 processor then you should request multiple cores using qlogin. This example won't work for launching graphical applications.

1. Login to Cheaha head node
<pre>
$ ssh cheaha.uabgrid.uab.edu
</pre>
2. Get access to a compute node
<pre>
$ qlogin -l h_rt=1:00:00,vf=1G -m b -M $USER@uab.edu -now no
</pre>
3. Following code snippet downloads and installs Python 3.3.1 in '$HOME/mypy'. It is provided as an example for type of work that can be done using a compute node. Any command line activity or code compilation can be done using a compute node.
<pre>
$ wget http://www.python.org/ftp/python/3.3.1/Python-3.3.1.tgz -P ~/tmp
$ tar xvzf ~/tmp/Python-3.3.1.tgz -C ~/src
$ pushd ~/src/Python-3.3.1
$ mkdir -p ~/mypy
$ ./configure --prefix ~/mypy
$ make
$ make install
</pre>
<pre>
$ ~/mypy/bin/python3 -V
Python 3.3.1
</pre>

== Code compilation or command line work using multiple cores ==

== Launching Graphical application ==
1. Login to Cheaha head node with X11 forwarding enabled
<pre>
$ ssh -X cheaha.uabgrid.uab.edu
</pre>
2. Get access to a compute node
<pre>
$ qlogin -l h_rt=1:00:00,vf=1G -m b -M $USER@uab.edu -now no
</pre>
3. Launching [http://www.broadinstitute.org/igv/ IGV] application (make sure you have it installed!).
<pre>
$ ~/apps/igv/igv.sh
1G
INFO [2013-04-09 14:59:40,392] [DirectoryManager.java:149] [main] IGV Directory: /home/pavgi/igv
INFO [2013-04-09 14:59:40,407] [Main.java:77] [main] Startup IGV Version 2.1.23 (2558)08/31/2012 11:27 PM
Fetching user directory... INFO [2013-04-09 14:59:40,898] [Main.java:78] [main] Default User Directory: /home/pavgi
INFO [2013-04-09 14:59:55,405] [GenomeManager.java:101] [main]
Loading genome: http://igvdata.broadinstitute.org/genomes/hg18.genome
INFO [2013-04-09 14:59:56,311] [GenomeManager.java:125] [main] Genome loaded. id= hg18
INFO [2013-04-09 14:59:59,224] [CommandListener.java:78] [Thread-7] Listening on port 60151
</pre>
[[File:IGV_qlogin_example.png|600px]]

== Using VNC session ==
When an SSH session is closed it's corresponding qlogin job will also get terminated. This can be frustrating when you need to keep your command line activity running, but need to close your main SSH session or turn-off your personal computer. It can solved by using a combination of SSH, VNC and qlogin. Please refer to our [[Setting_Up_VNC_Session|VNC]] setup guide to start a VNC session with Cheaha head node. After getting your VNC desktop, you can get a compute node resource using the qlogin command. A qlogin session launched through VNC won't get terminated even if your SSH or VNC window is closed. Of course, it will get terminated after specified run-time is reached.

Cheaha Quick Start Softwares

2013-04-09T20:47:02Z

Pavgi@uab.edu:

'''NOTE: This page is still under development. Please refer to [[Cheaha_GettingStarted]] page for detailed documentation.'''

= Software =
Cheaha's shared software stack includes many scientific computing softwares. Below is list of popular softwares available on Cheaha:
* [[Amber]]
* [[FFTW]]
* [[Gromacs]]
* [[GSL]]
* [[NAMD]]
* [[VMD]]
* [[Intel Compilers]]
* [[GNU Compilers]]
* [[Java]]
* [[R]]
* [[OpenMPI]]
* [[MATLAB]]

These softwares can be included in a job environment using [http://modules.sourceforge.net/ environment modules]. Environment modules make environment variables modification easy and repeatable. A complete list of shared software applications can be displayed using following modules command.
<pre>
$ module avail
</pre>

Most of our shared software applications are installed in '/share/apps' directory, but users can install software in their home directories or request a project specific software install install directory in '/share/apps' space. Please refer to guidelines below to install software in your home directory.

If you would like Research Computing to install a specific software in '/share/apps', then please send us an email to our [mailto:support@vo.uabgrid.uab.edu team] with installation details.

= Examples =
== Including software in a job environment ==
== Installing software in your home directory ==

Cheaha Quick Start Interactive Jobs

2013-04-09T20:45:17Z

Pavgi@uab.edu:

= Interactive Resources =

After you login to Cheaha the command-line interface that you see is running on the head node. Most of your light interactive prep-work to submit a compute job to the scheduler can be carried out on this head node. If you have a heavier workload to prepare for a batch job (eg. compiling code or other manipulations of data) or your compute application requires interactive control, you should request a dedicated interactive node for this work.

Interactive resources are requested by submitting an "interactive" job to the scheduler. Interactive jobs will provide you a command line on a compute resource that you can use just like you would the command line on the head node. The difference is that the scheduler has dedicated the requested resources to your job and you can run your interactive commands without having to worry about impacting other users on the head node.

Interactive jobs are requested with the qrsh or qlogin command (please use your correct email address in place of the ''$USER@uab.edu'' string if you do not have an @uab.edu email address):
<pre>
qlogin -l h_rt=1:00:00,vf=1G -m b -M $USER@uab.edu -now no
</pre>

This command requests 1 compute core with 1Gb of RAM for 1 hour. The command will wait until the resource is reserved by the scheduler and send you an email when the resource is available. The email alert can be useful during periods of heavy cluster demand when interactive resources reservations may have significant wait times.

= Examples =
Below are some examples on launching interactive jobs on Cheaha.

== Code compilation or command line work using single core ==
Following example uses default SSH options to login to Cheaha, launch interactive job to get a compute node and compile Python. This approach can be used to run any code compilation or perform any command line activity. If your actions are going to involve more than 1 processor then you should request multiple cores using qlogin. This example won't work for launching graphical applications.

1. Login to Cheaha head node
<pre>
$ ssh cheaha.uabgrid.uab.edu
</pre>
2. Get access to a compute node
<pre>
$ qlogin -l h_rt=1:00:00,vf=1G -m b -M $USER@uab.edu -now no
</pre>
3. Following code snippet downloads and installs Python 3.3.1 in '$HOME/mypy'. It is provided as an example for type of work that can be done using a compute node. Any command line activity or code compilation can be done using a compute node.
<pre>
$ wget http://www.python.org/ftp/python/3.3.1/Python-3.3.1.tgz -P ~/tmp
$ tar xvzf ~/tmp/Python-3.3.1.tgz -C ~/src
$ pushd ~/src/Python-3.3.1
$ mkdir -p ~/mypy
$ ./configure --prefix ~/mypy
$ make
$ make install
</pre>
<pre>
$ ~/mypy/bin/python3 -V
Python 3.3.1
</pre>

== Code compilation or command line work using multiple cores ==

== Launching Graphical application ==
1. Login to Cheaha head node with X11 forwarding enabled
<pre>
$ ssh -X cheaha.uabgrid.uab.edu
</pre>
2. Get access to a compute node
<pre>
$ qlogin -l h_rt=1:00:00,vf=1G -m b -M $USER@uab.edu -now no
</pre>
3. Launching [http://www.broadinstitute.org/igv/ IGV] application (make sure you have it installed!).
<pre>
$ ~/apps/igv/igv.sh
1G
INFO [2013-04-09 14:59:40,392] [DirectoryManager.java:149] [main] IGV Directory: /home/pavgi/igv
INFO [2013-04-09 14:59:40,407] [Main.java:77] [main] Startup IGV Version 2.1.23 (2558)08/31/2012 11:27 PM
Fetching user directory... INFO [2013-04-09 14:59:40,898] [Main.java:78] [main] Default User Directory: /home/pavgi
INFO [2013-04-09 14:59:55,405] [GenomeManager.java:101] [main]
Loading genome: http://igvdata.broadinstitute.org/genomes/hg18.genome
INFO [2013-04-09 14:59:56,311] [GenomeManager.java:125] [main] Genome loaded. id= hg18
INFO [2013-04-09 14:59:59,224] [CommandListener.java:78] [Thread-7] Listening on port 60151
</pre>
[[File:IGV_qlogin_example.png|600px]]

== Using VNC session ==
When an SSH session is closed it's corresponding qlogin job will also get terminated. This can be frustrating when you need to keep your command line activity running, but need to close your main SSH session or turn-off your personal computer. It can solved by using a combination of SSH, VNC and qlogin. Please refer to our [[Setting_Up_VNC_Session|VNC]] setup guide to start a VNC session with Cheaha head node. After getting your VNC desktop, you can get a compute node resource using the qlogin command. A qlogin session launched through VNC won't get terminated even if your SSH or VNC window is closed. Of course, it will get terminated after specified run-time is reached.

Cheaha Quick Start Softwares

2013-04-09T20:40:58Z

Pavgi@uab.edu:

= Software =
Cheaha's shared software stack includes many scientific computing softwares. Below is list of popular softwares available on Cheaha:
* [[Amber]]
* [[FFTW]]
* [[Gromacs]]
* [[GSL]]
* [[NAMD]]
* [[VMD]]
* [[Intel Compilers]]
* [[GNU Compilers]]
* [[Java]]
* [[R]]
* [[OpenMPI]]
* [[MATLAB]]

These softwares can be included in a job environment using [http://modules.sourceforge.net/ environment modules]. Environment modules make environment variables modification easy and repeatable. A complete list of shared software applications can be displayed using following modules command.
<pre>
$ module avail
</pre>

Most of our shared software applications are installed in '/share/apps' directory, but users can install software in their home directories or request a project specific software install install directory in '/share/apps' space. Please refer to guidelines below to install software in your home directory.

If you would like Research Computing to install a specific software in '/share/apps', then please send us an email to our [mailto:support@vo.uabgrid.uab.edu team] with installation details.

= Examples =
== Including software in a job environment ==
== Installing software in your home directory ==

Cheaha Quick Start Softwares

2013-04-09T20:39:03Z

Pavgi@uab.edu: added notes on available software

Cheaha Quick Start Interactive Jobs

2013-04-09T20:14:19Z

Pavgi@uab.edu: started page on interactive jobs and it's examples

= Interactive Resources =

After you login to Cheaha the command-line interface that you see is running on the head node. Most of your light interactive prep-work to submit a compute job to the scheduler can be carried out on this head node. If you have a heavier workload to prepare for a batch job (eg. compiling code or other manipulations of data) or your compute application requires interactive control, you should request a dedicated interactive node for this work.

Interactive resources are requested by submitting an "interactive" job to the scheduler. Interactive jobs will provide you a command line on a compute resource that you can use just like you would the command line on the head node. The difference is that the scheduler has dedicated the requested resources to your job and you can run your interactive commands without having to worry about impacting other users on the head node.

Interactive jobs are requested with the qrsh or qlogin command (please use your correct email address in place of the ''$USER@uab.edu'' string if you do not have an @uab.edu email address):
<pre>
qlogin -l h_rt=1:00:00,vf=1G -m b -M $USER@uab.edu -now no
</pre>

This command requests 1 compute core with 1Gb of RAM for 1 hour. The command will wait until the resource is reserved by the scheduler and send you an email when the resource is available. The email alert can be useful during periods of heavy cluster demand when interactive resources reservations may have significant wait times.

= Examples =
Below are some examples on launching interactive jobs on Cheaha.

== Code compilation or command line work using single core ==
Following example uses default SSH options to login to Cheaha, launch interactive job to get a compute node and compile Python. This approach can be used to run any code compilation or perform any command line activity. If your actions are going to involve more than 1 processor then you should request multiple cores using qlogin. This example won't work for launching graphical applications.

1. Login to Cheaha head node
<pre>
$ ssh cheaha.uabgrid.uab.edu
</pre>
2. Get access to a compute node
<pre>
$ qlogin -l h_rt=1:00:00,vf=1G -m b -M $USER@uab.edu -now no
</pre>
3. Following code snippet downloads and installs Python 3.3.1 in '$HOME/mypy'. It is provided as an example for type of work that can be done using a compute node. Any command line activity or code compilation can be done using a compute node.
<pre>
$ wget http://www.python.org/ftp/python/3.3.1/Python-3.3.1.tgz -P ~/tmp
$ tar xvzf ~/tmp/Python-3.3.1.tgz -C ~/src
$ pushd ~/src/Python-3.3.1
$ mkdir -p ~/mypy
$ ./configure --prefix ~/mypy
$ make
$ make install
</pre>
<pre>
$ ~/mypy/bin/python3 -V
Python 3.3.1
</pre>

== Code compilation or command line work using multiple cores ==

== Launching Graphical application ==
1. Login to Cheaha head node with X11 forwarding enabled
<pre>
$ ssh -X cheaha.uabgrid.uab.edu
</pre>
2. Get access to a compute node
$ qlogin -l h_rt=1:00:00,vf=1G -m b -M $USER@uab.edu -now no
</pre>
3. Launching [http://www.broadinstitute.org/igv/ IGV] application (make sure you have it installed!).
<pre>
$ ~/apps/igv/igv.sh
1G
INFO [2013-04-09 14:59:40,392] [DirectoryManager.java:149] [main] IGV Directory: /home/pavgi/igv
INFO [2013-04-09 14:59:40,407] [Main.java:77] [main] Startup IGV Version 2.1.23 (2558)08/31/2012 11:27 PM
Fetching user directory... INFO [2013-04-09 14:59:40,898] [Main.java:78] [main] Default User Directory: /home/pavgi
INFO [2013-04-09 14:59:55,405] [GenomeManager.java:101] [main]
Loading genome: http://igvdata.broadinstitute.org/genomes/hg18.genome
INFO [2013-04-09 14:59:56,311] [GenomeManager.java:125] [main] Genome loaded. id= hg18
INFO [2013-04-09 14:59:59,224] [CommandListener.java:78] [Thread-7] Listening on port 60151
</pre>
[[File:IGV_qlogin_example.png|600px]]

== Using VNC session ==
When an SSH session is closed it's corresponding qlogin job will also get terminated. This can be frustrating when you need to keep your command line activity running, but need to close your main SSH session or turn-off your personal computer. It can solved by using a combination of SSH, VNC and qlogin. Please refer to our [[Setting_Up_VNC_Session|VNC]] setup guide to start a VNC session with Cheaha head node. After getting your VNC desktop, you can get a compute node resource using the qlogin command. A qlogin session launched through VNC won't get terminated even if your SSH or VNC window is closed. Of course, it will get terminated after specified run-time is reached.

File:IGV qlogin example.png

2013-04-09T20:04:03Z

Pavgi@uab.edu:

Cheaha Quick Start (Deprecated)

2013-04-09T16:56:30Z

Pavgi@uab.edu: added note that page is under development

'''NOTE: This page is still under development. Please refer to [[Cheaha_GettingStarted]] page for detailed documentation.'''

Cheaha is a shared cluster computing environment for UAB researchers. Cheaha offers total 8.75 TFLOPS compute power, 200 TB high-performance storage and 2.8 TB memory. See [[Cheaha_Quick_Start_Hardware]] for more details on compute platform, but first let's get started with an example and see how easy it is to use.

If you have any questions about Cheaha usage then please contact Research Computing team at support@vo.uabgrid.uab.edu .

== Logging In ==
More [[Cheaha_GettingStarted#Login|detailed login instructions]] are also available.

Most users will authenticate to Cheaha using their BlazerID and associated password using an SSH (Secure Shell) client. The basic syntax is as follows:

<pre>
ssh BLAZERID@cheaha.uabgrid.uab.edu
</pre>

== Hello Cheaha! ==
A shared cluster environment like Cheaha uses a job scheduler to run tasks on the cluster to provide optimal resource sharing among users. Cheaha uses a job scheduling system call SGE to schedule and manage jobs. A user needs to tell SGE about resource requirements (e.g. CPU, memory) so that it can schedule jobs effectively. These resource requirements along with actual application code can be specified in a single file commonly referred as 'Job Script/File'. Following is a simple job script that prints job number and hostname.

<pre>
#!/bin/bash
#
# Define the shell used by your compute job
#$ -S /bin/bash
#
# Run in the current directory from where you submit the job
#$ -cwd
#
# Set the maximum runtime for the job (ex: 10 minutes)
#$ -l h_rt=00:10:00
# Set the maximum amount of RAM needed per slot (ex: 512 MB's)
#$ -l vf=512M
#
# Your email address
#$ -M YOUR_EMAIL_ADDRESS
#
# Notification Options:
# b Mail is sent at the beginning of the job
# e Mail is sent at the end of the job
# a Mail is sent when the job is aborted or rescheduled
# s Mail is sent when the job is suspended
# n No mail is sent
#$ -m eas
#
# Use the environment from your current shell
#
#$ -V

echo "The job $JOB_ID is running on $HOSTNAME"

</pre>

Lines starting with '#$' have a special meaning in the SGE world. SGE specific configuration options are specified after the '#$' characters. Above configuration options are useful for most job scripts and for additional configuration options refer to SGE commands manual. A job script is submitted to the cluster using SGE specific commands. There are many commands available, but following three commands are the most common:
* qsub - to submit job
* qdel - to delete job
* qstat - to view job status

We can submit above job script using qsub command:
<pre>
$ qsub HelloCheaha.sh
Your job 9043385 (HelloCheaha.sh) has been submitted
</pre>

When the job script is submitted, SGE queues it up and assigns it a job number (e.g. 9043385 in above example). The job number is available inside job script using environment variable $JOB_ID. This variable can be used inside job script to create job related directory structure or file names. [[Cheaha_Quick_Start_Job_Script_Examples]] provides more job script examples and [[Cheaha_Quick_Start_SGE_Commands]] provides more information about SGE commands.

== Software ==
Cheaha's software stack includes many scientific computing softwares. Below is list of popular softwares available on Cheaha:
* [[Amber]]
* [[FFTW]]
* [[Gromacs]]
* [[GSL]]
* [[NAMD]]
* [[VMD]]
* [[Intel Compilers]]
* [[GNU Compilers]]
* [[Java]]
* [[R]]
* [[OpenMPI]]
* [[MATLAB]]

These softwares can be included in a job environment using [http://modules.sourceforge.net/ environment modules]. Environment modules make environment variables modification easy and repeatable. Please refer to [[Cheaha_Quick_Start_Softwares]] page for more details.

== Storage ==
A non-trivial analysis requires a good storage backend that supports large file staging, access control and performance. Cheaha storage fabric includes a high-performance parallel file system called [http://wiki.lustre.org/index.php/Main_Page Lustre] which handles large files efficiently. It's available for all Cheaha users and specific details are covered on [[Cheaha_Quick_Start_Storage]] page.

== Graphical Interface ==
Some applications use graphical interface to perform certain actions (e.g. submit buttons, file selections etc.). Cheaha supports graphical applications using an interactive X-Windows session with SGE's qrsh command. This will allow you to run graphical applications like MATLAB or AFNI on Cheaha. Refer to [[Cheaha_Quick_Start_Interactive_Jobs]] for details on running graphical X-Windows applications.

== Scheduling Policies ==

== Support ==
If you have any questions about our documentation or need any help with Cheaha then please contact us on support@vo.uabgrid.uab.edu . Cheaha is maintained by [[About_Research_Computing|UAB IT's Research Computing team]].

Cheaha Quick Start (Deprecated)

2013-04-01T21:33:08Z

Pavgi@uab.edu: /* Support */ Added about research computing teaxt.

Cheaha is a shared cluster computing environment for UAB researchers. Cheaha offers total 8.75 TFLOPS compute power, 200 TB high-performance storage and 2.8 TB memory. See [[Cheaha_Quick_Start_Hardware]] for more details on compute platform, but first let's get started with an example and see how easy it is to use.

If you have any questions about Cheaha usage then please contact Research Computing team at support@vo.uabgrid.uab.edu .

== Hello Cheaha! ==
A shared cluster environment like Cheaha uses a job scheduler to run tasks on the cluster to provide optimal resource sharing among users. Cheaha uses a job scheduling system call SGE to schedule and manage jobs. A user needs to tell SGE about resource requirements (e.g. CPU, memory) so that it can schedule jobs effectively. These resource requirements along with actual application code can be specified in a single file commonly referred as 'Job Script/File'. Following is a simple job script that prints job number and hostname.

<pre>
#!/bin/bash
#
# Define the shell used by your compute job
#
#$ -S /bin/bash
#
# Tell the cluster to run in the current directory from where you submit the job
#
#$ -cwd
#
# Tell the scheduler only need 10 minutes
#
#$ -l h_rt=00:01:00,s_rt=0:00:55,vf=512M
#
# Set your email address and request notification when you job is complete or if it fails
#
#$ -M YOUR_EMAIL_ADDRESS
#$ -m eas
#
# Tell the scheduler to use the environment from your current shell
#
#$ -V

echo "The job $JOB_ID is running on $HOSTNAME"

</pre>

Lines starting with '#$' have a special meaning in the SGE world. SGE specific configuration options are specified after the '#$' characters. Above configuration options are useful for most job scripts and for additional configuration options refer to SGE commands manual. A job script is submitted to the cluster using SGE specific commands. There are many commands available, but following three commands are the most common:
* qsub - to submit job
* qdel - to delete job
* qstat - to view job status

We can submit above job script using qsub command:
<pre>
$ qsub HelloCheaha.sh
Your job 9043385 (HelloCheaha.sh) has been submitted
</pre>

When the job script is submitted, SGE queues it up and assigns it a job number (e.g. 9043385 in above example). The job number is available inside job script using environment variable $JOB_ID. This variable can be used inside job script to create job related directory structure or file names. [[Cheaha_Quick_Start_Job_Script_Examples]] provides more job script examples and [[Cheaha_Quick_Start_SGE_Commands]] provides more information about SGE commands.

== Software ==
Cheaha's software stack includes many scientific computing softwares. Below is list of popular softwares available on Cheaha:
* [[Amber]]
* [[FFTW]]
* [[Gromacs]]
* [[GSL]]
* [[NAMD]]
* [[VMD]]
* [[Intel Compilers]]
* [[GNU Compilers]]
* [[Java]]
* [[R]]
* [[OpenMPI]]
* [[MATLAB]]

These softwares can be included in a job environment using [http://modules.sourceforge.net/ environment modules]. Environment modules make environment variables modification easy and repeatable. Please refer to [[Cheaha_Quick_Start_Softwares]] page for more details.

== Storage ==
A non-trivial analysis requires a good storage backend that supports large file staging, access control and performance. Cheaha storage fabric includes a high-performance parallel file system called [http://wiki.lustre.org/index.php/Main_Page Lustre] which handles large files efficiently. It's available for all Cheaha users and specific details are covered on [[Cheaha_Quick_Start_Storage]] page.

== Graphical Interface ==
Some applications use graphical interface to perform certain actions (e.g. submit buttons, file selections etc.). Cheaha supports graphical applications using an interactive X-Windows session with SGE's qrsh command. This will allow you to run graphical applications like MATLAB or AFNI on Cheaha. Refer to [[Cheaha_Quick_Start_Interactive_Jobs]] for details on running graphical X-Windows applications.

== Scheduling Policies ==

== Support ==
If you have any questions about our documentation or need any help with Cheaha then please contact us on support@vo.uabgrid.uab.edu . Cheaha is maintained by [[About_Research_Computing|UAB IT's Research Computing team]].

Cheaha Quick Start (Deprecated)

2013-04-01T21:30:13Z

Pavgi@uab.edu: Added more sections

Cheaha Quick Start (Deprecated)

2013-03-27T19:49:17Z

Pavgi@uab.edu:

Cheaha is a shared cluster computing environment for UAB researchers. A shared cluster environment uses a job scheduler to run tasks on the cluster to provide optimal resource sharing among users. Cheaha uses a job scheduling system call SGE to schedule and manage jobs. A user needs to tell SGE about resource requirements (e.g. CPU, memory) so that it can schedule jobs effectively. These resource requirements along with actual application code can be specified in a single file commonly referred as 'Job Script/File'. Following is a simple job script that prints job number and hostname.

<pre>
#!/bin/bash
#
# Define the shell used by your compute job
#
#$ -S /bin/bash
#
# Tell the cluster to run in the current directory from where you submit the job
#
#$ -cwd
#
# Tell the scheduler only need 10 minutes
#
#$ -l h_rt=00:01:00,s_rt=0:00:55,vf=512M
#
# Set your email address and request notification when you job is complete or if it fails
#
#$ -M YOUR_EMAIL_ADDRESS
#$ -m eas
#
# Tell the scheduler to use the environment from your current shell
#
#$ -V

echo "The job $JOB_ID is running on $HOSTNAME"

</pre>

Lines starting with '#$' have a special meaning in the SGE world. SGE specific configuration options are specified after the '#$' characters. SGE offers many configuration options but above mentioned options should be useful for basic job scripts. A job script is submitted to the cluster using SGE specific commands. There are many commands available, but following three commands are the most common:
* qsub - to submit job
* qdel - to delete job
* qstat - to view job status

We can submit above job script using qsub command:
<pre>
$ qsub HelloCheaha.sh
Your job 9043385 (HelloCheaha.sh) has been submitted
</pre>

When the job script is submitted, SGE queues it up and assigns it a job number (e.g. 9043385 in above example). The job number is available inside job script using environment variable $JOB_ID. This variable can be used inside job script to create job related directory structure or file names.

Cheaha Quick Start (Deprecated)

2013-03-27T19:40:24Z

Pavgi@uab.edu:

Cheaha Quick Start (Deprecated)

2013-03-27T17:13:36Z

Pavgi@uab.edu: added common commands

Cheaha Quick Start (Deprecated)

2013-03-27T17:09:11Z

Pavgi@uab.edu: Created page with " Cheaha is a shared cluster computing environment for UAB researchers. A shared cluster environment uses a job scheduler to run tasks on the cluster to provide optimal resourc..."

Galaxy Importfs

2013-03-18T00:41:55Z

Pavgi@uab.edu: added limitation about compressed file uploads

UAB Galaxy can import data from a user's importfs directory on Cheaha cluster. This is a two step process as follows:
# Stage data files in importfs directory - /scratch/importfs/galaxy/$USER
# Select files that need to be uploaded into the galaxy

The first step of data staging in importfs directory happens outside of Galaxy. In the second step, a Galaxy user can select files in importfs staging directory and import them into Galaxy. When files in importfs staging directory are uploaded in Galaxy, their original version in importfs staging directory is deleted by Galaxy application.

Following documentation outlines steps involved in using UAB Galaxy's data import method.
== Account setup ==
# You need a Cheaha cluster account in order to transfer your data files to '/scratch/importfs/galaxy/$USER' directory. Please refer to [[Cheaha_GettingStarted#Access]] page for getting Cheaha account.
# Your '/scratch/importfs/galaxy/$USER' directory should get configured within 30-minutes after you get your Cheaha account.
# Make sure you can login to Cheaha cluster as documented [[Cheaha_GettingStarted#Login]].

== Data Transfer ==
Data transfer to Cheaha is briefly described in [[Cheaha_GettingStarted#Uploading_Data]]. Following examples cover Galaxy specific use cases to transfer data to '/scratch/importfs/galaxy/$USER' directory.
=== Pulling data from external URLs ===
A file accessible via external FTP or HTTP location can be downloaded using FTP or HTTP network download tools. Below is an example using wget network downloader tool which should work for both ftp and http URLs.
<pre>
# Change to Directory "/scratch/importfs/galaxy/$USER"
$ cd /scratch/importfs/galaxy/$USER
$ wget http://pavgi.uabgrid.uab.edu/seq.bam

</pre>
Note: wget has many other useful options which may help in downloading data. Refer to wget manual (command: man wget) to see all options.

=== Copying data to Cheaha using scp ===
[http://en.wikipedia.org/wiki/Secure_copy scp] is a file transfer protocol to securely copy files between two computer systems. Mac and Linux systems natively comes with scp support, however, Windows systems will need an external client like pscp or [http://winscp.net/eng/download.php#download2 WinSCP] (a friendly drag-and-drop app). Following example shows native scp client usage from Linux or Mac system. It should work in a similar manner with other scp clients.
<pre>
# scp <path-to-file-on-local-desktop> <BlazerID>@cheaha.uabgrid.uab.edu:/scratch/importfs/galaxy/<BlazerID>/
$ scp users.csv pavgi@cheaha.uabgrid.uab.edu:/scratch/importfs/galaxy/pavgi/
</pre>
Above command will transfer a file on a local desktop system to '/scratch/importfs/galaxy/<BlazerID>/' directory on Cheaha.

== Check file permissions ==
Make sure that transfered files have group read permissions so that galaxy can read them. In general, files have group read permissions set, so it won't be an issue in most cases.
<pre>
$ chmod g+r /scratch/importfs/galaxy/<BlazerID>/users.csv
</pre>

== Data import in Galaxy ==
Files deposited in '/scratch/importfs/galaxy/$USER' directory can be seen by Galaxy application and they are listed in data upload method as shown below.

[[File:FTP_Staged_Files.png|730px]]

Above example shows a single file staged in importfs directory, however, you can stage and import multiple files as well. When a file is successfully imported in Galaxy, it shows up in the right-side history panel and gets deleted from the importfs directory.

== Known Issues ==
The imortfs uploads are not working with compressed file formats such as fastq.gz. A compressed files will need to be uncompressed outside of Galaxy and then selecting uncompressed file(s) in the Galaxy upload form. We are working on resolving this issue.

Galaxy Importfs

2013-03-11T19:12:06Z

Pavgi@uab.edu: /* Check file permissions */

Galaxy Importfs

2013-03-11T18:41:41Z

Pavgi@uab.edu: added section on file permissions

UAB Galaxy can import data from a user's importfs directory on Cheaha cluster. This is a two step process as follows:
# Stage data files in importfs directory - /scratch/importfs/galaxy/$USER
# Select files that need to be uploaded into the galaxy

The first step of data staging in importfs directory happens outside of Galaxy. In the second step, a Galaxy user can select files in importfs staging directory and import them into Galaxy. When files in importfs staging directory are uploaded in Galaxy, their original version in importfs staging directory is deleted by Galaxy application.

Following documentation outlines steps involved in using UAB Galaxy's data import method.
== Account setup ==
# You need a Cheaha cluster account in order to transfer your data files to '/scratch/importfs/galaxy/$USER' directory. Please refer to [[Cheaha_GettingStarted#Access]] page for getting Cheaha account.
# Your '/scratch/importfs/galaxy/$USER' directory should get configured within 30-minutes after you get your Cheaha account.
# Make sure you can login to Cheaha cluster as documented [[Cheaha_GettingStarted#Login]].

== Data Transfer ==
Data transfer to Cheaha is briefly described in [[Cheaha_GettingStarted#Uploading_Data]]. Following examples cover Galaxy specific use cases to transfer data to '/scratch/importfs/galaxy/$USER' directory.
=== Pulling data from external URLs ===
A file accessible via external FTP or HTTP location can be downloaded using FTP or HTTP network download tools. Below is an example using wget network downloader tool which should work for both ftp and http URLs.
<pre>
# Change to Directory "/scratch/importfs/galaxy/$USER"
$ cd /scratch/importfs/galaxy/$USER
$ wget http://pavgi.uabgrid.uab.edu/seq.bam

</pre>
Note: wget has many other useful options which may help in downloading data. Refer to wget manual (command: man wget) to see all options.

=== Copying data to Cheaha using scp ===
[http://en.wikipedia.org/wiki/Secure_copy scp] is a file transfer protocol to securely copy files between two computer systems. Mac and Linux systems natively comes with scp support, however, Windows systems will need an external client like pscp or [http://winscp.net/eng/download.php#download2 WinSCP] (a friendly drag-and-drop app). Following example shows native scp client usage from Linux or Mac system. It should work in a similar manner with other scp clients.
<pre>
# scp <path-to-file-on-local-desktop> <BlazerID>@cheaha.uabgrid.uab.edu:/scratch/importfs/galaxy/<BlazerID>/
$ scp users.csv pavgi@cheaha.uabgrid.uab.edu:/scratch/importfs/galaxy/pavgi/
</pre>
Above command will transfer a file on a local desktop system to '/scratch/importfs/galaxy/<BlazerID>/' directory on Cheaha.

== Check file permissions ==
Make sure that transfered files have group write permissions so that galaxy can delete them.
<pre>
$ chmod g+w /scratch/importfs/galaxy/<BlazerID>/users.csv
</pre>

== Data import in Galaxy ==
Files deposited in '/scratch/importfs/galaxy/$USER' directory can be seen by Galaxy application and they are listed in data upload method as shown below.

[[File:FTP_Staged_Files.png|730px]]

Above example shows a single file staged in importfs directory, however, you can stage and import multiple files as well. When a file is successfully imported in Galaxy, it shows up in the right-side history panel and gets deleted from the importfs directory.

Galaxy Importfs

2013-03-11T16:53:11Z

Pavgi@uab.edu: /* Data import in Galaxy */

UAB Galaxy can import data from a user's importfs directory on Cheaha cluster. This is a two step process as follows:
# Stage data files in importfs directory - /scratch/importfs/galaxy/$USER
# Select files that need to be uploaded into the galaxy

The first step of data staging in importfs directory happens outside of Galaxy. In the second step, a Galaxy user can select files in importfs staging directory and import them into Galaxy. When files in importfs staging directory are uploaded in Galaxy, their original version in importfs staging directory is deleted by Galaxy application.

Following documentation outlines steps involved in using UAB Galaxy's data import method.
== Account setup ==
# You need a Cheaha cluster account in order to transfer your data files to '/scratch/importfs/galaxy/$USER' directory. Please refer to [[Cheaha_GettingStarted#Access]] page for getting Cheaha account.
# Your '/scratch/importfs/galaxy/$USER' directory should get configured within 30-minutes after you get your Cheaha account.
# Make sure you can login to Cheaha cluster as documented [[Cheaha_GettingStarted#Login]].

== Data Transfer ==
Data transfer to Cheaha is briefly described in [[Cheaha_GettingStarted#Uploading_Data]]. Following examples cover Galaxy specific use cases to transfer data to '/scratch/importfs/galaxy/$USER' directory.
=== Pulling data from external URLs ===
A file accessible via external FTP or HTTP location can be downloaded using FTP or HTTP network download tools. Below is an example using wget network downloader tool which should work for both ftp and http URLs.
<pre>
# Change to Directory "/scratch/importfs/galaxy/$USER"
$ cd /scratch/importfs/galaxy/$USER
$ wget http://pavgi.uabgrid.uab.edu/seq.bam

</pre>
Note: wget has many other useful options which may help in downloading data. Refer to wget manual (command: man wget) to see all options.

=== Copying data to Cheaha using scp ===
[http://en.wikipedia.org/wiki/Secure_copy scp] is a file transfer protocol to securely copy files between two computer systems. Mac and Linux systems natively comes with scp support, however, Windows systems will need an external client like pscp or WinSCP. Following example shows native scp client usage from Linux or Mac system. It should work in a similar manner with other scp clients.
<pre>
# scp <path-to-file-on-local-desktop> <BlazerID>@cheaha.uabgrid.uab.edu:/scratch/importfs/galaxy/<BlazerID>/
$ scp users.csv pavgi@cheaha.uabgrid.uab.edu:/scratch/importfs/galaxy/pavgi/
</pre>
Above command will transfer a file on a local desktop system to '/scratch/importfs/galaxy/<BlazerID>/' directory on Cheaha.

== Data import in Galaxy ==
Files deposited in '/scratch/importfs/galaxy/$USER' directory can be seen by Galaxy application and they are listed in data upload method as shown below.

[[File:FTP_Staged_Files.png|730px]]

Above example shows a single file staged in importfs directory, however, you can stage and import multiple files as well. When a file is successfully imported in Galaxy, it shows up in the right-side history panel and gets deleted from the importfs directory.

Galaxy Importfs

2013-03-11T16:46:58Z

Pavgi@uab.edu: /* Data Transfer */

UAB Galaxy can import data from a user's importfs directory on Cheaha cluster. This is a two step process as follows:
# Stage data files in importfs directory - /scratch/importfs/galaxy/$USER
# Select files that need to be uploaded into the galaxy

The first step of data staging in importfs directory happens outside of Galaxy. In the second step, a Galaxy user can select files in importfs staging directory and import them into Galaxy. When files in importfs staging directory are uploaded in Galaxy, their original version in importfs staging directory is deleted by Galaxy application.

Following documentation outlines steps involved in using UAB Galaxy's data import method.
== Account setup ==
# You need a Cheaha cluster account in order to transfer your data files to '/scratch/importfs/galaxy/$USER' directory. Please refer to [[Cheaha_GettingStarted#Access]] page for getting Cheaha account.
# Your '/scratch/importfs/galaxy/$USER' directory should get configured within 30-minutes after you get your Cheaha account.
# Make sure you can login to Cheaha cluster as documented [[Cheaha_GettingStarted#Login]].

== Data Transfer ==
Data transfer to Cheaha is briefly described in [[Cheaha_GettingStarted#Uploading_Data]]. Following examples cover Galaxy specific use cases to transfer data to '/scratch/importfs/galaxy/$USER' directory.
=== Pulling data from external URLs ===
A file accessible via external FTP or HTTP location can be downloaded using FTP or HTTP network download tools. Below is an example using wget network downloader tool which should work for both ftp and http URLs.
<pre>
# Change to Directory "/scratch/importfs/galaxy/$USER"
$ cd /scratch/importfs/galaxy/$USER
$ wget http://pavgi.uabgrid.uab.edu/seq.bam

</pre>
Note: wget has many other useful options which may help in downloading data. Refer to wget manual (command: man wget) to see all options.

=== Copying data to Cheaha using scp ===
[http://en.wikipedia.org/wiki/Secure_copy scp] is a file transfer protocol to securely copy files between two computer systems. Mac and Linux systems natively comes with scp support, however, Windows systems will need an external client like pscp or WinSCP. Following example shows native scp client usage from Linux or Mac system. It should work in a similar manner with other scp clients.
<pre>
# scp <path-to-file-on-local-desktop> <BlazerID>@cheaha.uabgrid.uab.edu:/scratch/importfs/galaxy/<BlazerID>/
$ scp users.csv pavgi@cheaha.uabgrid.uab.edu:/scratch/importfs/galaxy/pavgi/
</pre>
Above command will transfer a file on a local desktop system to '/scratch/importfs/galaxy/<BlazerID>/' directory on Cheaha.

== Data import in Galaxy ==
Files deposited in '/scratch/importfs/galaxy/$USER' directory can be seen by Galaxy application and they are listed in data upload method as shown below.

[[File:FTP_Staged_Files.png|730px]]

Above example shows only file staged in importfs directory, however, you can multiple files staged and selected for data import in Galaxy. All files successfully imported in Galaxy will show up right-side history panel and they will be deleted from importfs directory.

Galaxy Importfs

2013-03-11T16:38:26Z

Pavgi@uab.edu: /* Data import in Galaxy */

UAB Galaxy can import data from a user's importfs directory on Cheaha cluster. This is a two step process as follows:
# Stage data files in importfs directory - /scratch/importfs/galaxy/$USER
# Select files that need to be uploaded into the galaxy

The first step of data staging in importfs directory happens outside of Galaxy. In the second step, a Galaxy user can select files in importfs staging directory and import them into Galaxy. When files in importfs staging directory are uploaded in Galaxy, their original version in importfs staging directory is deleted by Galaxy application.

Following documentation outlines steps involved in using UAB Galaxy's data import method.
== Account setup ==
# You need a Cheaha cluster account in order to transfer your data files to '/scratch/importfs/galaxy/$USER' directory. Please refer to [[Cheaha_GettingStarted#Access]] page for getting Cheaha account.
# Your '/scratch/importfs/galaxy/$USER' directory should get configured within 30-minutes after you get your Cheaha account.
# Make sure you can login to Cheaha cluster as documented [[Cheaha_GettingStarted#Login]].

== Data Transfer ==
Data transfer to Cheaha is briefly described in [[Cheaha_GettingStarted#Uploading_Data]]. Following examples cover Galaxy specific use cases to transfer data in '/scratch/importfs/galaxy/$USER' directory.
=== Pulling data from external URLs ===
A file accessible via external FTP or HTTP location can be downloaded using FTP or HTTP network downloader tools. Below is an example using wget network downloader tool which should work for both ftp and http URLs.
<pre>
# Change to Directory "/scratch/importfs/galaxy/$USER"
$ cd /scratch/importfs/galaxy/$USER
$ wget http://pavgi.uabgrid.uab.edu/seq.bam

</pre>
Note, above is a simple wget example, wget has many other useful options and you can take go through them by reading it's manual page using 'man wget' command.

=== Copying data to Cheaha using scp ===
[http://en.wikipedia.org/wiki/Secure_copy scp] is a file transfer protocol to securely copy files between two computer systems. Mac and Linux systems natively comes with SCP support, however, for Windows system you will need to download external client like pscp or WinSCP. Following example shows native SCP application usage from Linux or Mac system. It should work in a similar manner with other SCP clients.
<pre>
# scp <path-to-file-on-local-desktop> <BlazerID>@cheaha.uabgrid.uab.edu:/scratch/importfs/galaxy/<BlazerID>/
$ scp users.csv pavgi@cheaha.uabgrid.uab.edu:/scratch/importfs/galaxy/pavgi/
</pre>
Above command will transfer a file on a local desktop system to '/scratch/importfs/galaxy/<BlazerID>/' directory on Cheaha.

== Data import in Galaxy ==
Files deposited in '/scratch/importfs/galaxy/$USER' directory can be seen by Galaxy application and they are listed in data upload method as shown below.

[[File:FTP_Staged_Files.png|730px]]

Above example shows only file staged in importfs directory, however, you can multiple files staged and selected for data import in Galaxy. All files successfully imported in Galaxy will show up right-side history panel and they will be deleted from importfs directory.

Galaxy Importfs

2013-03-08T17:08:54Z

Pavgi@uab.edu: /* Data Transfer */

UAB Galaxy can import data from a user's importfs directory on Cheaha cluster. This is a two step process as follows:
# Stage data files in importfs directory - /scratch/importfs/galaxy/$USER
# Select files that need to be uploaded into the galaxy

The first step of data staging in importfs directory happens outside of Galaxy. In the second step, a Galaxy user can select files in importfs staging directory and import them into Galaxy. When files in importfs staging directory are uploaded in Galaxy, their original version in importfs staging directory is deleted by Galaxy application.

Following documentation outlines steps involved in using UAB Galaxy's data import method.
== Account setup ==
# You need a Cheaha cluster account in order to transfer your data files to '/scratch/importfs/galaxy/$USER' directory. Please refer to [[Cheaha_GettingStarted#Access]] page for getting Cheaha account.
# Your '/scratch/importfs/galaxy/$USER' directory should get configured within 30-minutes after you get your Cheaha account.
# Make sure you can login to Cheaha cluster as documented [[Cheaha_GettingStarted#Login]].

== Data Transfer ==
Data transfer to Cheaha is briefly described in [[Cheaha_GettingStarted#Uploading_Data]]. Following examples cover Galaxy specific use cases to transfer data in '/scratch/importfs/galaxy/$USER' directory.
=== Pulling data from external URLs ===
A file accessible via external FTP or HTTP location can be downloaded using FTP or HTTP network downloader tools. Below is an example using wget network downloader tool which should work for both ftp and http URLs.
<pre>
# Change to Directory "/scratch/importfs/galaxy/$USER"
$ cd /scratch/importfs/galaxy/$USER
$ wget http://pavgi.uabgrid.uab.edu/seq.bam

</pre>
Note, above is a simple wget example, wget has many other useful options and you can take go through them by reading it's manual page using 'man wget' command.

=== Copying data to Cheaha using scp ===
[http://en.wikipedia.org/wiki/Secure_copy scp] is a file transfer protocol to securely copy files between two computer systems. Mac and Linux systems natively comes with SCP support, however, for Windows system you will need to download external client like pscp or WinSCP. Following example shows native SCP application usage from Linux or Mac system. It should work in a similar manner with other SCP clients.
<pre>
# scp <path-to-file-on-local-desktop> <BlazerID>@cheaha.uabgrid.uab.edu:/scratch/importfs/galaxy/<BlazerID>/
$ scp users.csv pavgi@cheaha.uabgrid.uab.edu:/scratch/importfs/galaxy/pavgi/
</pre>
Above command will transfer a file on a local desktop system to '/scratch/importfs/galaxy/<BlazerID>/' directory on Cheaha.

== Data import in Galaxy ==
Files deposited in '/scratch/importfs/galaxy/$USER' directory can be seen by Galaxy application and they are listed in data upload method as shown below.
[[File:FTP_Staged_Files.png|730px]]
Above example shows only file staged in importfs directory, however, you can multiple files staged and selected for data import in Galaxy. All files successfully imported in Galaxy will show up right-side history panel and they will be deleted from importfs directory.

Galaxy Importfs

2013-03-08T17:07:47Z

Pavgi@uab.edu: /* Account setup */

UAB Galaxy can import data from a user's importfs directory on Cheaha cluster. This is a two step process as follows:
# Stage data files in importfs directory - /scratch/importfs/galaxy/$USER
# Select files that need to be uploaded into the galaxy

The first step of data staging in importfs directory happens outside of Galaxy. In the second step, a Galaxy user can select files in importfs staging directory and import them into Galaxy. When files in importfs staging directory are uploaded in Galaxy, their original version in importfs staging directory is deleted by Galaxy application.

Following documentation outlines steps involved in using UAB Galaxy's data import method.
== Account setup ==
# You need a Cheaha cluster account in order to transfer your data files to '/scratch/importfs/galaxy/$USER' directory. Please refer to [[Cheaha_GettingStarted#Access]] page for getting Cheaha account.
# Your '/scratch/importfs/galaxy/$USER' directory should get configured within 30-minutes after you get your Cheaha account.
# Make sure you can login to Cheaha cluster as documented [[Cheaha_GettingStarted#Login]].

== Data Transfer ==
Data transfer to Cheaha is briefly described in Cheaha_GettingStarted#Uploading_Data. Following examples cover Galaxy specific use cases to transfer data in '/scratch/importfs/galaxy/$USER' directory.
=== Pulling data from external URLs ===
A file accessible via external FTP or HTTP location can be downloaded using FTP or HTTP network downloader tools. Below is an example using wget network downloader tool which should work for both ftp and http URLs.
<pre>
# Change to Directory "/scratch/importfs/galaxy/$USER"
$ cd /scratch/importfs/galaxy/$USER
$ wget http://pavgi.uabgrid.uab.edu/seq.bam

</pre>
Note, above is a simple wget example, wget has many other useful options and you can take go through them by reading it's manual page using 'man wget' command.

=== Copying data to Cheaha using SCP ===
[http://en.wikipedia.org/wiki/Secure_copy SCP] is a file transfer protocol to securely copy files between two computer systems. Mac and Linux systems natively comes with SCP support, however, for Windows system you will need to download external client like pscp or WinSCP. Following example shows native SCP application usage from Linux or Mac system. It should work in a similar manner with other SCP clients.
<pre>
# scp <path-to-file-on-local-desktop> <BlazerID>@cheaha.uabgrid.uab.edu:/scratch/importfs/galaxy/<BlazerID>/
$ scp users.csv pavgi@cheaha.uabgrid.uab.edu:/scratch/importfs/galaxy/pavgi/
</pre>
Above command will transfer a file on a local desktop system to '/scratch/importfs/galaxy/<BlazerID>/' directory on Cheaha.

== Data import in Galaxy ==
Files deposited in '/scratch/importfs/galaxy/$USER' directory can be seen by Galaxy application and they are listed in data upload method as shown below.
[[File:FTP_Staged_Files.png|730px]]
Above example shows only file staged in importfs directory, however, you can multiple files staged and selected for data import in Galaxy. All files successfully imported in Galaxy will show up right-side history panel and they will be deleted from importfs directory.

Galaxy Importfs

2013-03-08T17:06:53Z

Pavgi@uab.edu: updated image pixel size

UAB Galaxy can import data from a user's importfs directory on Cheaha cluster. This is a two step process as follows:
# Stage data files in importfs directory - /scratch/importfs/galaxy/$USER
# Select files that need to be uploaded into the galaxy

The first step of data staging in importfs directory happens outside of Galaxy. In the second step, a Galaxy user can select files in importfs staging directory and import them into Galaxy. When files in importfs staging directory are uploaded in Galaxy, their original version in importfs staging directory is deleted by Galaxy application.

Following documentation outlines steps involved in using UAB Galaxy's data import method.
== Account setup ==
# You need a Cheaha cluster account in order to transfer your data files to '/scratch/importfs/galaxy/$USER' directory. Please refer to Cheaha_GettingStarted#Access page for getting Cheaha account.
# Your '/scratch/importfs/galaxy/$USER' directory should get configured within 30-minutes after you get your Cheaha account.
# Make sure you can login to Cheaha cluster as documented [[Cheaha_GettingStarted#Login]].

== Data Transfer ==
Data transfer to Cheaha is briefly described in Cheaha_GettingStarted#Uploading_Data. Following examples cover Galaxy specific use cases to transfer data in '/scratch/importfs/galaxy/$USER' directory.
=== Pulling data from external URLs ===
A file accessible via external FTP or HTTP location can be downloaded using FTP or HTTP network downloader tools. Below is an example using wget network downloader tool which should work for both ftp and http URLs.
<pre>
# Change to Directory "/scratch/importfs/galaxy/$USER"
$ cd /scratch/importfs/galaxy/$USER
$ wget http://pavgi.uabgrid.uab.edu/seq.bam

</pre>
Note, above is a simple wget example, wget has many other useful options and you can take go through them by reading it's manual page using 'man wget' command.

=== Copying data to Cheaha using SCP ===
[http://en.wikipedia.org/wiki/Secure_copy SCP] is a file transfer protocol to securely copy files between two computer systems. Mac and Linux systems natively comes with SCP support, however, for Windows system you will need to download external client like pscp or WinSCP. Following example shows native SCP application usage from Linux or Mac system. It should work in a similar manner with other SCP clients.
<pre>
# scp <path-to-file-on-local-desktop> <BlazerID>@cheaha.uabgrid.uab.edu:/scratch/importfs/galaxy/<BlazerID>/
$ scp users.csv pavgi@cheaha.uabgrid.uab.edu:/scratch/importfs/galaxy/pavgi/
</pre>
Above command will transfer a file on a local desktop system to '/scratch/importfs/galaxy/<BlazerID>/' directory on Cheaha.

== Data import in Galaxy ==
Files deposited in '/scratch/importfs/galaxy/$USER' directory can be seen by Galaxy application and they are listed in data upload method as shown below.
[[File:FTP_Staged_Files.png|730px]]
Above example shows only file staged in importfs directory, however, you can multiple files staged and selected for data import in Galaxy. All files successfully imported in Galaxy will show up right-side history panel and they will be deleted from importfs directory.

Talk:Galaxy Data Import

2013-03-08T17:05:54Z

Pavgi@uab.edu: Pavgi@uab.edu moved page Talk:Galaxy Data Import to Talk:Galaxy Importfs: dropped word 'data import' to avoid confusion with data libraries

#REDIRECT [[Talk:Galaxy Importfs]]

Talk:Galaxy Importfs

2013-03-08T17:05:53Z

Pavgi@uab.edu: Pavgi@uab.edu moved page Talk:Galaxy Data Import to Talk:Galaxy Importfs: dropped word 'data import' to avoid confusion with data libraries

Should data transfer examples, especially wget and ftp-client, use compute nodes for pulling data? -- [[User:Pavgi@uab.edu]]

Galaxy Data Import

2013-03-08T17:05:53Z

Pavgi@uab.edu: Pavgi@uab.edu moved page Galaxy Data Import to Galaxy Importfs: dropped word 'data import' to avoid confusion with data libraries

#REDIRECT [[Galaxy Importfs]]

Galaxy Importfs

2013-03-08T17:05:53Z

Pavgi@uab.edu: Pavgi@uab.edu moved page Galaxy Data Import to Galaxy Importfs: dropped word 'data import' to avoid confusion with data libraries

UAB Galaxy can import data from a user's importfs directory on Cheaha cluster. This is a two step process as follows:
# Stage data files in importfs directory - /scratch/importfs/galaxy/$USER
# Select files that need to be uploaded into the galaxy

The first step of data staging in importfs directory happens outside of Galaxy. In the second step, a Galaxy user can select files in importfs staging directory and import them into Galaxy. When files in importfs staging directory are uploaded in Galaxy, their original version in importfs staging directory is deleted by Galaxy application.

Following documentation outlines steps involved in using UAB Galaxy's data import method.
== Account setup ==
# You need a Cheaha cluster account in order to transfer your data files to '/scratch/importfs/galaxy/$USER' directory. Please refer to Cheaha_GettingStarted#Access page for getting Cheaha account.
# Your '/scratch/importfs/galaxy/$USER' directory should get configured within 30-minutes after you get your Cheaha account.
# Make sure you can login to Cheaha cluster as documented [[Cheaha_GettingStarted#Login]].

== Data Transfer ==
Data transfer to Cheaha is briefly described in Cheaha_GettingStarted#Uploading_Data. Following examples cover Galaxy specific use cases to transfer data in '/scratch/importfs/galaxy/$USER' directory.
=== Pulling data from external URLs ===
A file accessible via external FTP or HTTP location can be downloaded using FTP or HTTP network downloader tools. Below is an example using wget network downloader tool which should work for both ftp and http URLs.
<pre>
# Change to Directory "/scratch/importfs/galaxy/$USER"
$ cd /scratch/importfs/galaxy/$USER
$ wget http://pavgi.uabgrid.uab.edu/seq.bam

</pre>
Note, above is a simple wget example, wget has many other useful options and you can take go through them by reading it's manual page using 'man wget' command.

=== Copying data to Cheaha using SCP ===
[http://en.wikipedia.org/wiki/Secure_copy SCP] is a file transfer protocol to securely copy files between two computer systems. Mac and Linux systems natively comes with SCP support, however, for Windows system you will need to download external client like pscp or WinSCP. Following example shows native SCP application usage from Linux or Mac system. It should work in a similar manner with other SCP clients.
<pre>
# scp <path-to-file-on-local-desktop> <BlazerID>@cheaha.uabgrid.uab.edu:/scratch/importfs/galaxy/<BlazerID>/
$ scp users.csv pavgi@cheaha.uabgrid.uab.edu:/scratch/importfs/galaxy/pavgi/
</pre>
Above command will transfer a file on a local desktop system to '/scratch/importfs/galaxy/<BlazerID>/' directory on Cheaha.

== Data import in Galaxy ==
Files deposited in '/scratch/importfs/galaxy/$USER' directory can be seen by Galaxy application and they are listed in data upload method as shown below.
[[File:FTP_Staged_Files.png]]
Above example shows only file staged in importfs directory, however, you can multiple files staged and selected for data import in Galaxy. All files successfully imported in Galaxy will show up right-side history panel and they will be deleted from importfs directory.

Galaxy File Uploads

2013-03-08T17:04:57Z

Pavgi@uab.edu:

[https://galaxy.uabgrid.uab.edu UAB Galaxy] supports data import in three ways:
# Direct file uploads to using a web browser
# Fetching data from external URLs through Galaxy
# Importing existing files on Cheaha file system

Web browser based file upload is a convenient approach, but not recommended for files larger than 2 GB in size because of browser limitations. Also, web browser based upload in Galaxy doesn't provide any feedback on upload progress and it can be an unreliable operation. Hence, it's recommended to stage data on Galaxy accessible file-system and then import it in Galaxy.

UAB Galaxy instance is configured to look for files in '/scratch/importfs/galaxy/$USER' and '/scratch/user/$USER' directories on Cheaha. Data files can be copied to Cheaha using [wikipedia:Secure_copy scp] or they can be downloaded using tools like wget, curl or ftp. Please refer to [[Cheaha_GettingStarted#Access]] page for getting access to Cheaha.

Following sections provide an overview of UAB Galaxy import methods.

# importfs or file drop-off mode: UAB Galaxy platform is configured to import files in $GALAXY_IMPORTFS directory on Cheaha (/scratch/importfs/galaxy/$USER). Galaxy application 'moves' files from imports directory to it's internal datasets directory. See [[Galaxy_Importfs]] page for more details on this upload method.
# Data Library: Galaxy has a concept of 'Data Libraries' which is a data container to organize files in an hierarchical manner, similar to directories on a desktop. Data libraries provide other features for data organization and sharing as well. Data libraries support files uploads using a web browser, fetching from external URLs and also by copying existing directories in a file-system. The file-system copy is similar to importfs option described above, however, it copies file to internal datasets directory rather than moving it. UAB Galaxy platform is configured to copy files in $USER_SCRATCH (/scratch/user/$USER) directory. See [[Galaxy_Data_Libraries]] page for more details on data libraries.

Galaxy Data Libraries

2013-03-08T16:39:07Z

Pavgi@uab.edu: /* Setting permissions on a sub-directory */

Galaxy has a concept of 'Data Libraries' which is a data container to organize files in an hierarchical manner, similar to directories on a desktop. Data libraries provide other features for data organization and sharing as well. You can add datasets to a data library by uploading them through a web browser, pasting external URL paths or referencing existing directory in $USER_SCRATCH on Cheaha.

== Creating a new data library ==
Right now Galaxy supports data library creation only for administrative users. You can request a new data library by sending an email to [mailto:galaxy-help@vo.uabgrid.uab.edu UAB Galaxy help list]. Please mention name, description and synopsis (optional) for your new data library. You will have full permissions to manage your data library access.
[[File:Managing_Data_Lib.png|730px]]

== Accessing a data library ==
A data library can be accessed by clicking 'Shared Data --> Data Libraries' item in the top menu bar.
[[File:Accessing_Data_Lib.png|730px]]

== Adding datasets to the data library ==
Data library provides three options to add datasets:
* Upload files
* Upload directory of files
* Import datasets in your current history

=== Upload files ===
The first option allows uploading files using a web browser and pasting external URL locations.
[[File:Uploading_Files_Data_Lib.png|730px]]

=== Upload directory of files ===
The second option lists existing directories in $USER_SCRATCH on Cheaha. Click on 'Server directory' drop-down menu to
see a list of files in your $USER_SCRATCH. Galaxy application needs read and browse permissions to use this upload option.
==== Setting permissions on $USER_SCRATCH ====
Galaxy application needs read and browse permissions on the $USER_SCRATCH and they can be set using Access Control List (ACL):
setfacl -m u:galaxy:rx /scratch/user/$USER
==== Setting permissions on a sub-directory ====
Galaxy application needs recursive read and browse permissions on a subdirectory that will be copied or linked. This can be done using following Access Control List (ACL) command:
setfacl -R -m u:galaxy:rx /lustre/scratch/$USER/demo-datasets

After setting appropriate read permissions for Galaxy application, the directory can be imported by selecting it in the drop-down menu and copying or linking it to the Galaxy.
[[File:Uploading_Dir_Of_Files_Data_Lib.png|730px]]

File:Uploading Dir Of Files Data Lib.png

2013-03-08T16:38:45Z

Pavgi@uab.edu:

Galaxy Data Libraries

2013-03-08T16:38:33Z

Pavgi@uab.edu: /* Setting permissions on a sub-directory */

Galaxy Data Libraries

2013-03-08T16:38:15Z

Pavgi@uab.edu:

Galaxy Data Libraries

2013-03-08T16:34:10Z

Pavgi@uab.edu: updated image pixel size

Galaxy Data Libraries

2013-03-08T16:33:30Z

Pavgi@uab.edu: /* Creating a new data library */

Galaxy has a concept of 'Data Libraries' which is a data container to organize files in an hierarchical manner, similar to directories on a desktop. Data libraries provide other features for data organization and sharing as well. You can add datasets to a data library by uploading them through a web browser, pasting external URL paths or referencing existing directory in $USER_SCRATCH on Cheaha.

== Creating a new data library ==
Right now Galaxy supports data library creation only for administrative users. You can request a new data library by sending an email to [mailto:galaxy-help@vo.uabgrid.uab.edu UAB Galaxy help list]. Please mention name, description and synopsis (optional) for your new data library. You will have full permissions to manage your data library access.
[[File:Managing_Data_Lib.png|730px]]

== Accessing a data library ==
A data library can be accessed by clicking 'Shared Data --> Data Libraries' item in the top menu bar.
[[File:Accessing_Data_Lib.png]]

== Adding datasets to the data library ==
Data library provides three options to add datasets:
* Upload files
* Upload directory of files
* Import datasets in your current history

=== Upload files ===
The first option allows uploading files using a web browser and pasting external URL locations.
[[File:Uploading_Files_Data_Lib.png]]

=== Upload directory of files ===
The second option lists existing directories in $USER_SCRATCH on Cheaha. Click on 'Server directory' drop-down menu to
see a list of files in your $USER_SCRATCH. Galaxy application needs read and browse permissions to use this upload option.
==== Setting permissions on $USER_SCRATCH ====
Galaxy application needs read and browse permissions on the $USER_SCRATCH and they can be set using Access Control List (ACL):
setfacl -m u:galaxy:rx /scratch/user/$USER
==== Setting permissions on a sub-directory ====
Galaxy application needs recursive read and browse permissions on a subdirectory that will be copied or linked. This can be done using following Access Control List (ACL) command:
setfacl -R -m u:galaxy:rx /lustre/scratch/$USER/demo-datasets

After setting appropriate read permissions for Galaxy application, the directory can be imported by selecting it in the drop-down menu and copying or linking it to the Galaxy.

File:Uploading Files Data Lib.png

2013-03-08T16:30:17Z

Pavgi@uab.edu:

File:Accessing Data Lib.png

2013-03-08T16:29:05Z

Pavgi@uab.edu:

File:Managing Data Lib.png

2013-03-08T16:28:36Z

Pavgi@uab.edu:

Galaxy Data Libraries

2013-03-08T16:27:44Z

Pavgi@uab.edu: initial draft for data library uploads

Galaxy has a concept of 'Data Libraries' which is a data container to organize files in an hierarchical manner, similar to directories on a desktop. Data libraries provide other features for data organization and sharing as well. You can add datasets to a data library by uploading them through a web browser, pasting external URL paths or referencing existing directory in $USER_SCRATCH on Cheaha.

== Creating a new data library ==
Right now Galaxy supports data library creation only for administrative users. You can request a new data library by sending an email to [mailto:galaxy-help@vo.uabgrid.uab.edu UAB Galaxy help list]. Please mention name, description and synopsis (optional) for your new data library. You will have full permissions to manage your data library access.
[[File:Managing_Data_Lib.png]]

== Accessing a data library ==
A data library can be accessed by clicking 'Shared Data --> Data Libraries' item in the top menu bar.
[[File:Accessing_Data_Lib.png]]

== Adding datasets to the data library ==
Data library provides three options to add datasets:
* Upload files
* Upload directory of files
* Import datasets in your current history

=== Upload files ===
The first option allows uploading files using a web browser and pasting external URL locations.
[[File:Uploading_Files_Data_Lib.png]]

=== Upload directory of files ===
The second option lists existing directories in $USER_SCRATCH on Cheaha. Click on 'Server directory' drop-down menu to
see a list of files in your $USER_SCRATCH. Galaxy application needs read and browse permissions to use this upload option.
==== Setting permissions on $USER_SCRATCH ====
Galaxy application needs read and browse permissions on the $USER_SCRATCH and they can be set using Access Control List (ACL):
setfacl -m u:galaxy:rx /scratch/user/$USER
==== Setting permissions on a sub-directory ====
Galaxy application needs recursive read and browse permissions on a subdirectory that will be copied or linked. This can be done using following Access Control List (ACL) command:
setfacl -R -m u:galaxy:rx /lustre/scratch/$USER/demo-datasets

After setting appropriate read permissions for Galaxy application, the directory can be imported by selecting it in the drop-down menu and copying or linking it to the Galaxy.

Galaxy File Uploads

2013-03-08T02:44:44Z

Pavgi@uab.edu: updated page name from Galaxy_Data_Library_Import to Galaxy_Data_Libraries

[https://galaxy.uabgrid.uab.edu UAB Galaxy] supports data import in three ways:
# Direct file uploads to using a web browser
# Fetching data from external URLs through Galaxy
# Importing existing files on Cheaha file system

Web browser based file upload is a convenient approach, but not recommended for files larger than 2 GB in size because of browser limitations. Also, web browser based upload in Galaxy doesn't provide any feedback on upload progress and it can be an unreliable operation. Hence, it's recommended to stage data on Galaxy accessible file-system and then import it in Galaxy.

UAB Galaxy instance is configured to look for files in '/scratch/importfs/galaxy/$USER' and '/scratch/user/$USER' directories on Cheaha. Data files can be copied to Cheaha using [wikipedia:Secure_copy scp] or they can be downloaded using tools like wget, curl or ftp. Please refer to [[Cheaha_GettingStarted#Access]] page for getting access to Cheaha.

Following sections provide an overview of UAB Galaxy import methods.

# importfs or file drop-off mode: UAB Galaxy platform is configured to import files in '/scratch/importfs/galaxy/$USER' directory. Galaxy application 'moves' files from imports directory to it's internal datasets directory. See [[Galaxy_Data_Import]] page for more details on the importfs method.
# Data Library: Galaxy has a concept of 'Data Libraries' which is a data container to organize files in an hierarchical manner, similar to directories on a desktop. Data libraries provide other features for data organization and sharing as well. Data libraries support direct files uploads from a web browser, data fetch from external URLs and also file system imports. The file system import options is similar to importfs option described above, however, it copies file to internal datasets directory rather than moving it. UAB Galaxy platform is configured to import files in '/scratch/user/$USER' directory. See [[Galaxy_Data_Libraries]] page for more details on this data import method.

Galaxy File Uploads

2013-03-08T02:42:48Z

Pavgi@uab.edu:

[https://galaxy.uabgrid.uab.edu UAB Galaxy] supports data import in three ways:
# Direct file uploads to using a web browser
# Fetching data from external URLs through Galaxy
# Importing existing files on Cheaha file system

Web browser based file upload is a convenient approach, but not recommended for files larger than 2 GB in size because of browser limitations. Also, web browser based upload in Galaxy doesn't provide any feedback on upload progress and it can be an unreliable operation. Hence, it's recommended to stage data on Galaxy accessible file-system and then import it in Galaxy.

UAB Galaxy instance is configured to look for files in '/scratch/importfs/galaxy/$USER' and '/scratch/user/$USER' directories on Cheaha. Data files can be copied to Cheaha using [wikipedia:Secure_copy scp] or they can be downloaded using tools like wget, curl or ftp. Please refer to [[Cheaha_GettingStarted#Access]] page for getting access to Cheaha.

Following sections provide an overview of UAB Galaxy import methods.

# importfs or file drop-off mode: UAB Galaxy platform is configured to import files in '/scratch/importfs/galaxy/$USER' directory. Galaxy application 'moves' files from imports directory to it's internal datasets directory. See [[Galaxy_Data_Import]] page for more details on the importfs method.
# Data Library: Galaxy has a concept of 'Data Libraries' which is a data container to organize files in an hierarchical manner, similar to directories on a desktop. Data libraries provide other features for data organization and sharing as well. Data libraries support direct files uploads from a web browser, data fetch from external URLs and also file system imports. The file system import options is similar to importfs option described above, however, it copies file to internal datasets directory rather than moving it. UAB Galaxy platform is configured to import files in '/scratch/user/$USER' directory. See [[Galaxy_Data_Library_Import]] page for more details on this data import method.

Galaxy File Uploads

2013-03-08T02:15:22Z

Pavgi@uab.edu: updated ftp related to sections with importfs and revised other sections aswell

[https://galaxy.uabgrid.uab.edu UAB Galaxy] supports data import in three ways:
# Direct file uploads to using a web browser
# Fetching data from external URLs through Galaxy
# Importing existing files on Cheaha file system

Web browser based file upload is a convenient approach, but not recommended for files larger than 2 GB in size because of browser limitations. Also, web browser based upload in Galaxy doesn't provide any feedback on upload progress and it can be an unreliable operation. You can fetch data from external URL locations through Galaxy, but that's not an efficient operation. Hence, it's recommended to stage data on Galaxy accessible file-system and then import it in Galaxy.

UAB Galaxy instance is configured to look for files in '/scratch/importfs/galaxy/$USER' and '/scratch/user/$USER' directories on Cheaha. Data files can be copied to Cheaha using [wikipedia:Secure_copy scp] or they can be downloaded using tools like wget, curl or ftp. Please refer to [[Cheaha_GettingStarted#Access]] page for getting access to Cheaha.

Following sections provide an overview of UAB Galaxy import methods.

# importfs or file drop-off mode: UAB Galaxy platform is configured to import files in '/scratch/importfs/galaxy/$USER' directory. Galaxy application 'moves' files from imports directory to it's internal datasets directory. See [[Galaxy_Data_Import]] page for more details on the importfs method.
# Data Library: Galaxy has a concept of 'Data Libraries' which is a data container to organize files in an hierarchical manner, similar to directories on a desktop. Data libraries provide other features for data organization and sharing as well. Data libraries support direct files uploads from a web browser, data fetch from external URLs and also file system imports. The file system import options is similar to FTP option described above, however, it doesn't delete original file on the file system after it has been imported in Galaxy. UAB Galaxy platform is configured to import files in '/scratch/user/$USER' directory. See [[Galaxy_Data_Library_Import]] page for more details on this data import method.

Galaxy Importfs

2013-03-07T19:37:42Z

Pavgi@uab.edu:

UAB Galaxy can import data from a user's importfs directory on Cheaha cluster. This is a two step process as follows:
# Stage data files in importfs directory - /scratch/importfs/galaxy/$USER
# Select files that need to be uploaded into the galaxy

The first step of data staging in importfs directory happens outside of Galaxy. In the second step, a Galaxy user can select files in importfs staging directory and import them into Galaxy. When files in importfs staging directory are uploaded in Galaxy, their original version in importfs staging directory is deleted by Galaxy application.

Following documentation outlines steps involved in using UAB Galaxy's data import method.
== Account setup ==
# You need a Cheaha cluster account in order to transfer your data files to '/scratch/importfs/galaxy/$USER' directory. Please refer to Cheaha_GettingStarted#Access page for getting Cheaha account.
# Your '/scratch/importfs/galaxy/$USER' directory should get configured within 30-minutes after you get your Cheaha account.
# Make sure you can login to Cheaha cluster as documented [[Cheaha_GettingStarted#Login]].

== Data Transfer ==
Data transfer to Cheaha is briefly described in Cheaha_GettingStarted#Uploading_Data. Following examples cover Galaxy specific use cases to transfer data in '/scratch/importfs/galaxy/$USER' directory.
=== Pulling data from external URLs ===
A file accessible via external FTP or HTTP location can be downloaded using FTP or HTTP network downloader tools. Below is an example using wget network downloader tool which should work for both ftp and http URLs.
<pre>
# Change to Directory "/scratch/importfs/galaxy/$USER"
$ cd /scratch/importfs/galaxy/$USER
$ wget http://pavgi.uabgrid.uab.edu/seq.bam

</pre>
Note, above is a simple wget example, wget has many other useful options and you can take go through them by reading it's manual page using 'man wget' command.

=== Copying data to Cheaha using SCP ===
[http://en.wikipedia.org/wiki/Secure_copy SCP] is a file transfer protocol to securely copy files between two computer systems. Mac and Linux systems natively comes with SCP support, however, for Windows system you will need to download external client like pscp or WinSCP. Following example shows native SCP application usage from Linux or Mac system. It should work in a similar manner with other SCP clients.
<pre>
# scp <path-to-file-on-local-desktop> <BlazerID>@cheaha.uabgrid.uab.edu:/scratch/importfs/galaxy/<BlazerID>/
$ scp users.csv pavgi@cheaha.uabgrid.uab.edu:/scratch/importfs/galaxy/pavgi/
</pre>
Above command will transfer a file on a local desktop system to '/scratch/importfs/galaxy/<BlazerID>/' directory on Cheaha.

== Data import in Galaxy ==
Files deposited in '/scratch/importfs/galaxy/$USER' directory can be seen by Galaxy application and they are listed in data upload method as shown below.
[[File:FTP_Staged_Files.png]]
Above example shows only file staged in importfs directory, however, you can multiple files staged and selected for data import in Galaxy. All files successfully imported in Galaxy will show up right-side history panel and they will be deleted from importfs directory.

Galaxy Importfs

2013-03-07T19:37:02Z

Pavgi@uab.edu: Removed FTP word to avoid confusion

UAB Galaxy can import data from a user's importfs directory on Cheaha cluster. This is a two step process as follows:
# Stage data files in importfs directory - /scratch/importfs/galaxy/$USER
# Select files that need to be uploaded into the galaxy

The first step of data staging in importfs directory happens outside of Galaxy. In the second step, a Galaxy user can select files in importfs staging directory and import them into Galaxy. When files in importfs staging directory are uploaded in Galaxy, their original version in importfs staging directory is deleted by Galaxy application.

Following documentation outlines steps involved in using Galaxy's 'FTP upload' method.
== Account setup ==
# You need a Cheaha cluster account in order to transfer your data files to '/scratch/importfs/galaxy/$USER' directory. Please refer to Cheaha_GettingStarted#Access page for getting Cheaha account.
# Your '/scratch/importfs/galaxy/$USER' directory should get configured within 30-minutes after you get your Cheaha account.
# Make sure you can login to Cheaha cluster as documented [[Cheaha_GettingStarted#Login]].

== Data Transfer ==
Data transfer to Cheaha is briefly described in Cheaha_GettingStarted#Uploading_Data. Following examples cover Galaxy specific use cases to transfer data in '/scratch/importfs/galaxy/$USER' directory.
=== Pulling data from external URLs ===
A file accessible via external FTP or HTTP location can be downloaded using FTP or HTTP network downloader tools. Below is an example using wget network downloader tool which should work for both ftp and http URLs.
<pre>
# Change to Directory "/scratch/importfs/galaxy/$USER"
$ cd /scratch/importfs/galaxy/$USER
$ wget http://pavgi.uabgrid.uab.edu/seq.bam

</pre>
Note, above is a simple wget example, wget has many other useful options and you can take go through them by reading it's manual page using 'man wget' command.

=== Copying data to Cheaha using SCP ===
[http://en.wikipedia.org/wiki/Secure_copy SCP] is a file transfer protocol to securely copy files between two computer systems. Mac and Linux systems natively comes with SCP support, however, for Windows system you will need to download external client like pscp or WinSCP. Following example shows native SCP application usage from Linux or Mac system. It should work in a similar manner with other SCP clients.
<pre>
# scp <path-to-file-on-local-desktop> <BlazerID>@cheaha.uabgrid.uab.edu:/scratch/importfs/galaxy/<BlazerID>/
$ scp users.csv pavgi@cheaha.uabgrid.uab.edu:/scratch/importfs/galaxy/pavgi/
</pre>
Above command will transfer a file on a local desktop system to '/scratch/importfs/galaxy/<BlazerID>/' directory on Cheaha.

== Data import in Galaxy ==
Files deposited in '/scratch/importfs/galaxy/$USER' directory can be seen by Galaxy application and they are listed in data upload method as shown below.
[[File:FTP_Staged_Files.png]]
Above example shows only file staged in importfs directory, however, you can multiple files staged and selected for data import in Galaxy. All files successfully imported in Galaxy will show up right-side history panel and they will be deleted from importfs directory.

Talk:Galaxy FTP Upload

2013-03-07T18:07:53Z

Pavgi@uab.edu: Pavgi@uab.edu moved page Talk:Galaxy FTP Upload to Talk:Galaxy Data Import

#REDIRECT [[Talk:Galaxy Data Import]]