Galaxy File Uploads - Revision history

Tanthony@uab.edu: added section privacy with Sensitive information template

2018-05-03T15:49:41Z

added section privacy with Sensitive information template

← Older revision		Revision as of 15:49, 3 May 2018
Line 1:		Line 1:
	~~{{SensitiveInformation}}~~
	[https://galaxy.uabgrid.uab.edu UAB Galaxy] supports data import in three ways:		[https://galaxy.uabgrid.uab.edu UAB Galaxy] supports data import in three ways:

Line 16:		Line 15:
	\|-		\|-
	\|}		\|}

			==Privacy==
			{{SensitiveInformation}}

	==Direct file uploads to using a web browser==		==Direct file uploads to using a web browser==

Tanthony@uab.edu: added Sensitiveinformation Template

2018-05-03T15:25:26Z

added Sensitiveinformation Template

← Older revision		Revision as of 15:25, 3 May 2018
Line 1:		Line 1:
			{{SensitiveInformation}}
	[https://galaxy.uabgrid.uab.edu UAB Galaxy] supports data import in three ways:		[https://galaxy.uabgrid.uab.edu UAB Galaxy] supports data import in three ways:

Curtish@uab.edu: remove chmod warning for importfs

2013-03-11T19:04:55Z

remove chmod warning for importfs

← Older revision		Revision as of 19:04, 11 March 2013
Line 12:		Line 12:
	\|-		\|-
	\| Importing files via the Cheaha file system		\| Importing files via the Cheaha file system
	\| requires an [[Cheaha_GettingStarted#Access\|account]] on cheaha, ~~must use cmd~~-line ~~to [[Galaxy_Importfs#Check_file_permissions\|edit permissions]]~~		\| requires an [[Cheaha_GettingStarted#Access\|account]] on cheaha, but command-line can be avoided
	\|-		\|-
	\|}		\|}

Curtish@uab.edu: add warning about permissions.

2013-03-11T18:56:20Z

add warning about permissions.

← Older revision		Revision as of 18:56, 11 March 2013
Line 12:		Line 12:
	\|-		\|-
	\| Importing files via the Cheaha file system		\| Importing files via the Cheaha file system
	\| requires an [[Cheaha_GettingStarted#Access\|account]] on cheaha		\| requires an [[Cheaha_GettingStarted#Access\|account]] on cheaha, must use cmd-line to [[Galaxy_Importfs#Check_file_permissions\|edit permissions]]
	\|-		\|-
	\|}		\|}

Curtish@uab.edu: added section headers, link to WinSCP, fixed wikipedia link for SCP

2013-03-11T18:01:01Z

added section headers, link to WinSCP, fixed wikipedia link for SCP

← Older revision		Revision as of 18:01, 11 March 2013
Line 9:		Line 9:
	\|-		\|-
	\| Fetching data from external URLs through Galaxy (ftp/http)		\| Fetching data from external URLs through Galaxy (ftp/http)
	\| can't access some password protected sites		\| can't access some password protected sites, such as the HudsonAlpha GSL
	\|-		\|-
	\| Importing ~~existing~~ files on Cheaha file system		\| Importing files via the Cheaha file system
	\| requires an account on cheaha		\| requires an [[Cheaha_GettingStarted#Access\|account]] on cheaha
	\|-		\|-
	\|}		\|}

			==Direct file uploads to using a web browser==
	Web browser based file upload is a convenient approach, but not recommended for files larger than 2 GB in size because of browser limitations. Also, web browser based upload in Galaxy doesn't provide any feedback on upload progress and it can be an unreliable operation. Hence, it's recommended to stage data on Galaxy accessible file-system and then import it in Galaxy.		Web browser based file upload is a convenient approach, but not recommended for files larger than 2 GB in size because of browser limitations. Also, web browser based upload in Galaxy doesn't provide any feedback on upload progress and it can be an unreliable operation. Hence, it's recommended to stage data on Galaxy accessible file-system and then import it in Galaxy.

	UAB Galaxy instance is configured to look for files in '/scratch/importfs/galaxy/$USER' and '/scratch/user/$USER' directories on Cheaha. Data files can be copied to Cheaha using [~~wikipedia~~:Secure_copy scp] or they can be downloaded using tools like wget, curl or ftp. Please refer to [[Cheaha_GettingStarted#Access]] page for getting access to Cheaha.		==Importing files via the Cheaha file system==
			UAB Galaxy instance is configured to look for files in '/scratch/importfs/galaxy/$USER' and '/scratch/user/$USER' directories on Cheaha. Data files can be copied to Cheaha using [[Wikipedia:Secure_copy\|scp]] or they can be downloaded using tools like wget, curl or ftp. A nice windows-friendly drag-and-drop tool is [http://winscp.net/eng/download.php#download2 WinSCP]. Please refer to [[Cheaha_GettingStarted#Access]] page for getting access to Cheaha.

	Following sections provide an overview of UAB Galaxy import methods.		Following sections provide an overview of UAB Galaxy import methods.

Curtish@uab.edu: turn number list to table to add list of drawbacks/limits

2013-03-11T17:53:49Z

turn number list to table to add list of drawbacks/limits

← Older revision		Revision as of 17:53, 11 March 2013
Line 1:		Line 1:
	[https://galaxy.uabgrid.uab.edu UAB Galaxy] supports data import in three ways:		[https://galaxy.uabgrid.uab.edu UAB Galaxy] supports data import in three ways:
	# Direct file uploads to using a web browser
	# Fetching data from external URLs through Galaxy		{\| border="1"
	# Importing existing files on Cheaha file system		\|+
			! Method !! Limitation
			\|-
			\| Direct file uploads to using a web browser
			\| only files < 2G
			\|-
			\| Fetching data from external URLs through Galaxy (ftp/http)
			\| can't access some password protected sites
			\|-
			\| Importing existing files on Cheaha file system
			\| requires an account on cheaha
			\|-
			\|}

	Web browser based file upload is a convenient approach, but not recommended for files larger than 2 GB in size because of browser limitations. Also, web browser based upload in Galaxy doesn't provide any feedback on upload progress and it can be an unreliable operation. Hence, it's recommended to stage data on Galaxy accessible file-system and then import it in Galaxy.		Web browser based file upload is a convenient approach, but not recommended for files larger than 2 GB in size because of browser limitations. Also, web browser based upload in Galaxy doesn't provide any feedback on upload progress and it can be an unreliable operation. Hence, it's recommended to stage data on Galaxy accessible file-system and then import it in Galaxy.

Pavgi@uab.edu at 17:04, 8 March 2013

2013-03-08T17:04:57Z

← Older revision		Revision as of 17:04, 8 March 2013
Line 10:		Line 10:
	Following sections provide an overview of UAB Galaxy import methods.		Following sections provide an overview of UAB Galaxy import methods.

	# importfs or file drop-off mode: UAB Galaxy platform is configured to import files in '/scratch/importfs/galaxy/$USER~~' directory~~. Galaxy application 'moves' files from imports directory to it's internal datasets directory. See [[~~Galaxy_Data_Import~~]] page for more details on ~~the importfs~~ method.		# importfs or file drop-off mode: UAB Galaxy platform is configured to import files in $GALAXY_IMPORTFS directory on Cheaha (/scratch/importfs/galaxy/$USER). Galaxy application 'moves' files from imports directory to it's internal datasets directory. See [[Galaxy_Importfs]] page for more details on this upload method.
	# Data Library: Galaxy has a concept of 'Data Libraries' which is a data container to organize files in an hierarchical manner, similar to directories on a desktop. Data libraries provide other features for data organization and sharing as well. Data libraries support ~~direct~~ files uploads ~~from~~ a web browser, ~~data fetch~~ from external URLs and also file system ~~imports~~. The file system ~~import options~~ is similar to importfs option described above, however, it copies file to internal datasets directory rather than moving it. UAB Galaxy platform is configured to ~~import~~ files in '/scratch/user/$USER' directory. See [[Galaxy_Data_Libraries]] page for more details on ~~this~~ data ~~import method~~.		# Data Library: Galaxy has a concept of 'Data Libraries' which is a data container to organize files in an hierarchical manner, similar to directories on a desktop. Data libraries provide other features for data organization and sharing as well. Data libraries support files uploads using a web browser, fetching from external URLs and also by copying existing directories in a file-system. The file-system copy is similar to importfs option described above, however, it copies file to internal datasets directory rather than moving it. UAB Galaxy platform is configured to copy files in $USER_SCRATCH (/scratch/user/$USER) directory. See [[Galaxy_Data_Libraries]] page for more details on data libraries.

Pavgi@uab.edu: updated page name from Galaxy_Data_Library_Import to Galaxy_Data_Libraries

2013-03-08T02:44:44Z

updated page name from Galaxy_Data_Library_Import to Galaxy_Data_Libraries

← Older revision		Revision as of 02:44, 8 March 2013
Line 11:		Line 11:

	# importfs or file drop-off mode: UAB Galaxy platform is configured to import files in '/scratch/importfs/galaxy/$USER' directory. Galaxy application 'moves' files from imports directory to it's internal datasets directory. See [[Galaxy_Data_Import]] page for more details on the importfs method.		# importfs or file drop-off mode: UAB Galaxy platform is configured to import files in '/scratch/importfs/galaxy/$USER' directory. Galaxy application 'moves' files from imports directory to it's internal datasets directory. See [[Galaxy_Data_Import]] page for more details on the importfs method.
	# Data Library: Galaxy has a concept of 'Data Libraries' which is a data container to organize files in an hierarchical manner, similar to directories on a desktop. Data libraries provide other features for data organization and sharing as well. Data libraries support direct files uploads from a web browser, data fetch from external URLs and also file system imports. The file system import options is similar to importfs option described above, however, it copies file to internal datasets directory rather than moving it. UAB Galaxy platform is configured to import files in '/scratch/user/$USER' directory. See [[~~Galaxy_Data_Library_Import~~]] page for more details on this data import method.		# Data Library: Galaxy has a concept of 'Data Libraries' which is a data container to organize files in an hierarchical manner, similar to directories on a desktop. Data libraries provide other features for data organization and sharing as well. Data libraries support direct files uploads from a web browser, data fetch from external URLs and also file system imports. The file system import options is similar to importfs option described above, however, it copies file to internal datasets directory rather than moving it. UAB Galaxy platform is configured to import files in '/scratch/user/$USER' directory. See [[Galaxy_Data_Libraries]] page for more details on this data import method.

Pavgi@uab.edu at 02:42, 8 March 2013

2013-03-08T02:42:48Z

← Older revision		Revision as of 02:42, 8 March 2013
Line 4:		Line 4:
	# Importing existing files on Cheaha file system		# Importing existing files on Cheaha file system

	Web browser based file upload is a convenient approach, but not recommended for files larger than 2 GB in size because of browser limitations. Also, web browser based upload in Galaxy doesn't provide any feedback on upload progress and it can be an unreliable ~~operation. You can fetch data from external URL locations through Galaxy, but that's not an efficient~~ operation. Hence, it's recommended to stage data on Galaxy accessible file-system and then import it in Galaxy.		Web browser based file upload is a convenient approach, but not recommended for files larger than 2 GB in size because of browser limitations. Also, web browser based upload in Galaxy doesn't provide any feedback on upload progress and it can be an unreliable operation. Hence, it's recommended to stage data on Galaxy accessible file-system and then import it in Galaxy.

	UAB Galaxy instance is configured to look for files in '/scratch/importfs/galaxy/$USER' and '/scratch/user/$USER' directories on Cheaha. Data files can be copied to Cheaha using [wikipedia:Secure_copy scp] or they can be downloaded using tools like wget, curl or ftp. Please refer to [[Cheaha_GettingStarted#Access]] page for getting access to Cheaha.		UAB Galaxy instance is configured to look for files in '/scratch/importfs/galaxy/$USER' and '/scratch/user/$USER' directories on Cheaha. Data files can be copied to Cheaha using [wikipedia:Secure_copy scp] or they can be downloaded using tools like wget, curl or ftp. Please refer to [[Cheaha_GettingStarted#Access]] page for getting access to Cheaha.
Line 11:		Line 11:

	# importfs or file drop-off mode: UAB Galaxy platform is configured to import files in '/scratch/importfs/galaxy/$USER' directory. Galaxy application 'moves' files from imports directory to it's internal datasets directory. See [[Galaxy_Data_Import]] page for more details on the importfs method.		# importfs or file drop-off mode: UAB Galaxy platform is configured to import files in '/scratch/importfs/galaxy/$USER' directory. Galaxy application 'moves' files from imports directory to it's internal datasets directory. See [[Galaxy_Data_Import]] page for more details on the importfs method.
	# Data Library: Galaxy has a concept of 'Data Libraries' which is a data container to organize files in an hierarchical manner, similar to directories on a desktop. Data libraries provide other features for data organization and sharing as well. Data libraries support direct files uploads from a web browser, data fetch from external URLs and also file system imports. The file system import options is similar to ~~FTP~~ option described above, however, it ~~doesn't delete original~~ file ~~on the file system after~~ it ~~has been imported in Galaxy~~. UAB Galaxy platform is configured to import files in '/scratch/user/$USER' directory. See [[Galaxy_Data_Library_Import]] page for more details on this data import method.		# Data Library: Galaxy has a concept of 'Data Libraries' which is a data container to organize files in an hierarchical manner, similar to directories on a desktop. Data libraries provide other features for data organization and sharing as well. Data libraries support direct files uploads from a web browser, data fetch from external URLs and also file system imports. The file system import options is similar to importfs option described above, however, it copies file to internal datasets directory rather than moving it. UAB Galaxy platform is configured to import files in '/scratch/user/$USER' directory. See [[Galaxy_Data_Library_Import]] page for more details on this data import method.

Pavgi@uab.edu: updated ftp related to sections with importfs and revised other sections aswell

2013-03-08T02:15:22Z

updated ftp related to sections with importfs and revised other sections aswell

← Older revision		Revision as of 02:15, 8 March 2013
Line 4:		Line 4:
	# Importing existing files on Cheaha file system		# Importing existing files on Cheaha file system

	Web browser based file upload is a convenient approach, but not recommended for files larger than 1 GB in size because of browser limitations. Also, web browser based upload in Galaxy doesn't provide any feedback on upload progress and it can be an unreliable operation. You can fetch data from external URL locations through Galaxy, but that's not an efficient operation. Hence, it's recommended to ~~use the last option where files can be imported from [[Cheaha]]~~ file system~~. You can transfer files to Cheaha using tools like [http://en.wikipedia.org/wiki/Secure_copy SCP]~~ and ~~later~~ import ~~these files~~ in ~~Galaxy application. You will need an account on Cheaha cluster to transfer files to it. Please refer to~~		Web browser based file upload is a convenient approach, but not recommended for files larger than 2 GB in size because of browser limitations. Also, web browser based upload in Galaxy doesn't provide any feedback on upload progress and it can be an unreliable operation. You can fetch data from external URL locations through Galaxy, but that's not an efficient operation. Hence, it's recommended to stage data on Galaxy accessible file-system and then import it in Galaxy.
	~~[[Cheaha_GettingStarted#Access]] page for getting an account on it. Following sections provides an overview of methods to import existing Cheaha files into~~ Galaxy.

	Galaxy ~~provides two options~~ to ~~import data on the filesystem:~~		UAB Galaxy instance is configured to look for files in '/scratch/importfs/galaxy/$USER' and '/scratch/user/$USER' directories on Cheaha. Data files can be copied to Cheaha using [wikipedia:Secure_copy scp] or they can be downloaded using tools like wget, curl or ftp. Please refer to [[Cheaha_GettingStarted#Access]] page for getting access to Cheaha.
	~~# FTP or file drop-off mode: Galaxy provides FTP upload option~~ in ~~the UI to import files from a~~ user'~~s FTP directory~~. ~~Although Galaxy call this FTP upload method, it doesn't really require any FTP setup to upload~~ files~~. The 'FTP upload' option should~~ be ~~considered as 'FTP~~ or ~~file drop-off' type mode, where files dropped in a directory~~ can be ~~'moved' into~~ Galaxy ~~application~~. ~~When Galaxy imports files from 'FTP~~ or drop-off~~' directory, the original file in is deleted. On~~ UAB Galaxy platform ~~we have~~ configured ~~this directory as~~ '/~~lustre~~/importfs/galaxy/$USER' ~~and you can get your~~ files ~~in this~~ directory ~~using scp, wget or any other transfer mechanisms~~. See [[~~Galaxy_FTP_Upload~~]] page for more details on ~~this data import~~ method.
	# Data Library: Galaxy has a concept of 'Data Libraries' which ~~provide~~ a data container to organize ~~your data~~ in a hierarchical manner, similar to directories on a desktop. Data libraries provide other features for data organization and sharing as well. Data libraries support direct files uploads from a web browser, data fetch from external URLs and also file system imports. The file system import options is similar to FTP option described above, however, it doesn't delete original file on the file system after it has been imported in Galaxy. On UAB Galaxy platform ~~we have~~ configured ~~Galaxy~~ to import files in '~~/lustre~~/scratch/~~$USER' directory if the~~ user ~~configures appropriate permissions for Galaxy application. You can get your files in '/lustre/scratch~~/$USER' directory ~~using scp, wget or any other transfer mechanisms~~. See [[Galaxy_Data_Library_Import]] page for more details on this data import method.		Following sections provide an overview of UAB Galaxy import methods.

			# importfs or file drop-off mode: UAB Galaxy platform is configured to import files in '/scratch/importfs/galaxy/$USER' directory. Galaxy application 'moves' files from imports directory to it's internal datasets directory. See [[Galaxy_Data_Import]] page for more details on the importfs method.
			# Data Library: Galaxy has a concept of 'Data Libraries' which is a data container to organize files in an hierarchical manner, similar to directories on a desktop. Data libraries provide other features for data organization and sharing as well. Data libraries support direct files uploads from a web browser, data fetch from external URLs and also file system imports. The file system import options is similar to FTP option described above, however, it doesn't delete original file on the file system after it has been imported in Galaxy. UAB Galaxy platform is configured to import files in '/scratch/user/$USER' directory. See [[Galaxy_Data_Library_Import]] page for more details on this data import method.