Galaxy Importfs
https://docs.rc.uab.edu/
Please use the new documentation url https://docs.rc.uab.edu/ for all Research Computing documentation needs.
As a result of this move, we have deprecated use of this wiki for documentation. We are providing read-only access to the content to facilitate migration of bookmarks and to serve as an historical record. All content updates should be made at the new documentation site. The original wiki will not receive further updates.
Thank you,
The Research Computing Team
UAB Galaxy can import data from a user's importfs directory on Cheaha cluster. This is a two step process as follows:
- Stage data files in importfs directory - /scratch/importfs/galaxy/$USER
- Select files that need to be uploaded into the galaxy
The first step of data staging in importfs directory happens outside of Galaxy. In the second step, a Galaxy user can select files in importfs staging directory and import them into Galaxy. When files in importfs staging directory are uploaded in Galaxy, their original version in importfs staging directory is deleted by Galaxy application.
Following documentation outlines steps involved in using UAB Galaxy's data import method.
Account setup
- You need a Cheaha cluster account in order to transfer your data files to '/scratch/importfs/galaxy/$USER' directory. Please refer to Cheaha_GettingStarted#Access page for getting Cheaha account.
- Your '/scratch/importfs/galaxy/$USER' directory should get configured within 30-minutes after you get your Cheaha account.
- Make sure you can login to Cheaha cluster as documented Cheaha_GettingStarted#Login.
Data Transfer
Data transfer to Cheaha is briefly described in Cheaha_GettingStarted#Uploading_Data. Following examples cover Galaxy specific use cases to transfer data in '/scratch/importfs/galaxy/$USER' directory.
Pulling data from external URLs
A file accessible via external FTP or HTTP location can be downloaded using FTP or HTTP network downloader tools. Below is an example using wget network downloader tool which should work for both ftp and http URLs.
# Change to Directory "/scratch/importfs/galaxy/$USER" $ cd /scratch/importfs/galaxy/$USER $ wget http://pavgi.uabgrid.uab.edu/seq.bam
Note, above is a simple wget example, wget has many other useful options and you can take go through them by reading it's manual page using 'man wget' command.
Copying data to Cheaha using SCP
SCP is a file transfer protocol to securely copy files between two computer systems. Mac and Linux systems natively comes with SCP support, however, for Windows system you will need to download external client like pscp or WinSCP. Following example shows native SCP application usage from Linux or Mac system. It should work in a similar manner with other SCP clients.
# scp <path-to-file-on-local-desktop> <BlazerID>@cheaha.uabgrid.uab.edu:/scratch/importfs/galaxy/<BlazerID>/ $ scp users.csv pavgi@cheaha.uabgrid.uab.edu:/scratch/importfs/galaxy/pavgi/
Above command will transfer a file on a local desktop system to '/scratch/importfs/galaxy/<BlazerID>/' directory on Cheaha.
Data import in Galaxy
Files deposited in '/scratch/importfs/galaxy/$USER' directory can be seen by Galaxy application and they are listed in data upload method as shown below.
Above example shows only file staged in importfs directory, however, you can multiple files staged and selected for data import in Galaxy. All files successfully imported in Galaxy will show up right-side history panel and they will be deleted from importfs directory.