Galaxy Importfs
https://docs.rc.uab.edu/
Please use the new documentation url https://docs.rc.uab.edu/ for all Research Computing documentation needs.
As a result of this move, we have deprecated use of this wiki for documentation. We are providing read-only access to the content to facilitate migration of bookmarks and to serve as an historical record. All content updates should be made at the new documentation site. The original wiki will not receive further updates.
Thank you,
The Research Computing Team
UAB Galaxy can import data from a user's importfs directory on Cheaha cluster. This is a two step process as follows:
- Stage data files in importfs directory - /scratch/importfs/galaxy/$USER
- Select files that need to be uploaded into the galaxy
The first step of data staging in importfs directory happens outside of Galaxy. In the second step, a Galaxy user can select files in importfs staging directory and import them into Galaxy. When files in importfs staging directory are uploaded in Galaxy, their original version in importfs staging directory is deleted by Galaxy application.
Following documentation outlines steps involved in using UAB Galaxy's data import method.
Account setup
- You need a Cheaha cluster account in order to transfer your data files to '/scratch/importfs/galaxy/$USER' directory. Please refer to Cheaha_GettingStarted#Access page for getting Cheaha account.
- Your '/scratch/importfs/galaxy/$USER' directory should get configured within 30-minutes after you get your Cheaha account.
- Make sure you can login to Cheaha cluster as documented Cheaha_GettingStarted#Login.
Data Transfer
Data transfer to Cheaha is briefly described in Cheaha_GettingStarted#Uploading_Data. Following examples cover Galaxy specific use cases to transfer data to '/scratch/importfs/galaxy/$USER' directory.
Pulling data from external URLs
A file accessible via external FTP or HTTP location can be downloaded using FTP or HTTP network download tools. Below is an example using wget network downloader tool which should work for both ftp and http URLs.
# Change to Directory "/scratch/importfs/galaxy/$USER" $ cd /scratch/importfs/galaxy/$USER $ wget http://pavgi.uabgrid.uab.edu/seq.bam
Note: wget has many other useful options which may help in downloading data. Refer to wget manual (command: man wget) to see all options.
Copying data to Cheaha using scp
scp is a file transfer protocol to securely copy files between two computer systems. Mac and Linux systems natively comes with scp support, however, Windows systems will need an external client like pscp or WinSCP (a friendly drag-and-drop app). Following example shows native scp client usage from Linux or Mac system. It should work in a similar manner with other scp clients.
# scp <path-to-file-on-local-desktop> <BlazerID>@cheaha.uabgrid.uab.edu:/scratch/importfs/galaxy/<BlazerID>/ $ scp users.csv pavgi@cheaha.uabgrid.uab.edu:/scratch/importfs/galaxy/pavgi/
Above command will transfer a file on a local desktop system to '/scratch/importfs/galaxy/<BlazerID>/' directory on Cheaha.
Check file permissions
Make sure that transfered files have group read permissions so that galaxy can read them. In general, files have group read permissions set, so it won't be an issue in most cases.
$ chmod g+r /scratch/importfs/galaxy/<BlazerID>/users.csv
Data import in Galaxy
Files deposited in '/scratch/importfs/galaxy/$USER' directory can be seen by Galaxy application and they are listed in data upload method as shown below.
Above example shows a single file staged in importfs directory, however, you can stage and import multiple files as well. When a file is successfully imported in Galaxy, it shows up in the right-side history panel and gets deleted from the importfs directory.
Known Issues
The imortfs uploads are not working with compressed file formats such as fastq.gz. A compressed files will need to be uncompressed outside of Galaxy and then selecting uncompressed file(s) in the Galaxy upload form. We are working on resolving this issue.