Galaxy File Uploads

From UABgrid Documentation
(Difference between revisions)
Jump to: navigation, search
(turn number list to table to add list of drawbacks/limits)
(added section headers, link to WinSCP, fixed wikipedia link for SCP)
Line 9: Line 9:
 
|-
 
|-
 
|  Fetching data from external URLs through Galaxy (ftp/http)
 
|  Fetching data from external URLs through Galaxy (ftp/http)
|  can't access some password protected sites
+
|  can't access some password protected sites, such as the HudsonAlpha GSL
 
|-
 
|-
|  Importing existing files on Cheaha file system
+
|  Importing files via the Cheaha file system
|  requires an account on cheaha
+
|  requires an [[Cheaha_GettingStarted#Access|account]] on cheaha
 
|-
 
|-
 
|}
 
|}
  
 +
==Direct file uploads to using a web browser==
 
Web browser based file upload is a convenient approach, but not recommended for files larger than 2 GB in size because of browser limitations. Also, web browser based upload in Galaxy doesn't provide any feedback on upload progress and it can be an unreliable operation. Hence, it's recommended to stage data on Galaxy accessible file-system and then import it in Galaxy.
 
Web browser based file upload is a convenient approach, but not recommended for files larger than 2 GB in size because of browser limitations. Also, web browser based upload in Galaxy doesn't provide any feedback on upload progress and it can be an unreliable operation. Hence, it's recommended to stage data on Galaxy accessible file-system and then import it in Galaxy.
  
UAB Galaxy instance is configured to look for files in '/scratch/importfs/galaxy/$USER' and '/scratch/user/$USER' directories on Cheaha. Data files can be copied to Cheaha using [wikipedia:Secure_copy scp] or they can be downloaded using tools like wget, curl or ftp. Please refer to [[Cheaha_GettingStarted#Access]] page for getting access to Cheaha.
+
==Importing files via the Cheaha file system==
 +
UAB Galaxy instance is configured to look for files in '/scratch/importfs/galaxy/$USER' and '/scratch/user/$USER' directories on Cheaha. Data files can be copied to Cheaha using [[Wikipedia:Secure_copy|scp]] or they can be downloaded using tools like wget, curl or ftp. A nice windows-friendly drag-and-drop tool is [http://winscp.net/eng/download.php#download2 WinSCP]. Please refer to [[Cheaha_GettingStarted#Access]] page for getting access to Cheaha.
  
 
Following sections provide an overview of UAB Galaxy import methods.  
 
Following sections provide an overview of UAB Galaxy import methods.  

Revision as of 12:01, 11 March 2013

UAB Galaxy supports data import in three ways:

Method Limitation
Direct file uploads to using a web browser only files < 2G
Fetching data from external URLs through Galaxy (ftp/http) can't access some password protected sites, such as the HudsonAlpha GSL
Importing files via the Cheaha file system requires an account on cheaha

Direct file uploads to using a web browser

Web browser based file upload is a convenient approach, but not recommended for files larger than 2 GB in size because of browser limitations. Also, web browser based upload in Galaxy doesn't provide any feedback on upload progress and it can be an unreliable operation. Hence, it's recommended to stage data on Galaxy accessible file-system and then import it in Galaxy.

Importing files via the Cheaha file system

UAB Galaxy instance is configured to look for files in '/scratch/importfs/galaxy/$USER' and '/scratch/user/$USER' directories on Cheaha. Data files can be copied to Cheaha using scp or they can be downloaded using tools like wget, curl or ftp. A nice windows-friendly drag-and-drop tool is WinSCP. Please refer to Cheaha_GettingStarted#Access page for getting access to Cheaha.

Following sections provide an overview of UAB Galaxy import methods.

  1. importfs or file drop-off mode: UAB Galaxy platform is configured to import files in $GALAXY_IMPORTFS directory on Cheaha (/scratch/importfs/galaxy/$USER). Galaxy application 'moves' files from imports directory to it's internal datasets directory. See Galaxy_Importfs page for more details on this upload method.
  2. Data Library: Galaxy has a concept of 'Data Libraries' which is a data container to organize files in an hierarchical manner, similar to directories on a desktop. Data libraries provide other features for data organization and sharing as well. Data libraries support files uploads using a web browser, fetching from external URLs and also by copying existing directories in a file-system. The file-system copy is similar to importfs option described above, however, it copies file to internal datasets directory rather than moving it. UAB Galaxy platform is configured to copy files in $USER_SCRATCH (/scratch/user/$USER) directory. See Galaxy_Data_Libraries page for more details on data libraries.
Personal tools
Namespaces

Variants
Actions
Navigation
Toolbox