UploadLargeData - Revision history

Curtish@uab.edu: removed page. Left links to other related pages. It was redundent.

2013-03-07T19:37:11Z

removed page. Left links to other related pages. It was redundent.

← Older revision		Revision as of 19:37, 7 March 2013
Line 1:		Line 1:
	= ~~Load and Link approach~~ =		= Page removed =

			Please see [[Galaxy_File_Uploads]] for an overview and [[Galaxy_Data_Import]] for large datasets.
	~~== transfer and uncompress (slow) ==~~
	~~# login to cheaha.uabgrid.uab.edu (linux),~~
	~~# create directory for this data set in your scratch dir~~
	~~## mkdir /lustre/scratch/''user''/proj1~~
	~~## make sure that directory is readable by galaxy user~~
	~~### Chmod og+x /lustre/scratch/''user''~~
	~~### Chmod og+x /lustre/scratch/''user''/proj1~~
	~~# transfer the files with SCP~~
	~~## scp *.fastq.gz ''user''@cheaha.uabgrid.uab.edu:/lustre/scratch/''user''/proj1~~
	~~## I used "Secure Shell Client" for windows, available at~~ [~~http://www.uab.edu/it/software/displaytitle.php?ItemID=34 UABIT~~]
	~~## open source client is [http://www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY~~]
	~~# '''UNCOMPRESS fastq.gz files!!'''~~
	~~## cd /lustre/scratch/''user''/proj1~~
	~~### gzip -d ''filename''~~
	~~### !!WARNING: the following parallel decompress stesp fail~~ for ~~docs over 8G uncompressed!!~~
	~~### find `pwd` -name "*.gz" -exec ksh -c 'qrsh "gzip -d \{}" &' \;~~
	~~### ls -1 *.gz \| xargs -L 1 -i_f_ ksh -c 'qrsh -cwd gzip -d _f_ &' \;~~
	~~# make sure the files are readable by galaxy~~
	~~## if you're in galaxy-admin UNIX group you can do~~
	~~### chgrp galaxy-admin *.fastq~~
	~~### chmod g+r *.fastq~~
	~~## if you're not, then you have to make it readable to the world (o=other)~~
	### chmod o+r /lustre/scratch/''user''/proj1/*

	~~== link into galaxy dataset (fast) ==~~
	~~# get Admin privileges on galaxy~~
	~~## either get Shantanu to make you admin in Galaxy~~
	~~## or grab someone who is (John, Curtis)~~
	~~# In Galaxy GUI:~~
	~~## admin > Manage Data Libraries > create new library~~
	~~### add Datasets~~
	~~#### Upload option: Upload files from system path~~
	~~##### Get a list of absolute path names using one of the following~~
	~~###### cd /lustre/scratch/''user''/proj1 THEN RUN find `pwd` -name "*.fastq"~~
	~~###### find /lustre/scratch/''user''/proj1 -name "*.fastq"~~
	~~##### paste list of absolute path names into URL/Text box in Web Admin GUI~~
	~~#### Change "Copy data into Galaxy?" to "Link to files without copying into Galaxy"~~
	~~#### Put something mnemonic in Message box.~~
	[[~~File:UploadLinkDataFilesGUI.jpg~~]]

	~~== link data into a history (fast) ==~~

	~~I could then select the~~ datasets ~~and, at bottom of page "For selected datasets: <Import to histories>" and get them into a history so I can compute on them~~.

Ozborn@uab.edu: Added image

2011-09-13T17:54:17Z

Added image

← Older revision		Revision as of 17:54, 13 September 2011
Line 40:		Line 40:
	#### Change "Copy data into Galaxy?" to "Link to files without copying into Galaxy"		#### Change "Copy data into Galaxy?" to "Link to files without copying into Galaxy"
	#### Put something mnemonic in Message box.		#### Put something mnemonic in Message box.
			[[File:UploadLinkDataFilesGUI.jpg]]

	== link data into a history (fast) ==		== link data into a history (fast) ==

	I could then select the datasets and, at bottom of page "For selected datasets: <Import to histories>" and get them into a history so I can compute on them.		I could then select the datasets and, at bottom of page "For selected datasets: <Import to histories>" and get them into a history so I can compute on them.

Curtish@uab.edu at 15:56, 27 July 2011

2011-07-27T15:56:09Z

← Older revision		Revision as of 15:56, 27 July 2011
Line 15:		Line 15:
	# '''UNCOMPRESS fastq.gz files!!'''		# '''UNCOMPRESS fastq.gz files!!'''
	## cd /lustre/scratch/''user''/proj1		## cd /lustre/scratch/''user''/proj1
			### gzip -d ''filename''
			### !!WARNING: the following parallel decompress stesp fail for docs over 8G uncompressed!!
	### find `pwd` -name "*.gz" -exec ksh -c 'qrsh "gzip -d \{}" &' \;		### find `pwd` -name "*.gz" -exec ksh -c 'qrsh "gzip -d \{}" &' \;
	### ls -1 *.gz \| xargs -L 1 -i_f_ ksh -c 'qrsh -cwd gzip -d _f_ &' \;		### ls -1 *.gz \| xargs -L 1 -i_f_ ksh -c 'qrsh -cwd gzip -d _f_ &' \;
	~~### gzip -d ''filename''~~
	# make sure the files are readable by galaxy		# make sure the files are readable by galaxy
	## if you're in galaxy-admin UNIX group you can do		## if you're in galaxy-admin UNIX group you can do

Curtish@uab.edu: /* transfer and uncompress (slow) */

2011-07-08T20:01:27Z

transfer and uncompress (slow)

← Older revision		Revision as of 20:01, 8 July 2011
Line 18:		Line 18:
	### ls -1 *.gz \| xargs -L 1 -i_f_ ksh -c 'qrsh -cwd gzip -d _f_ &' \;		### ls -1 *.gz \| xargs -L 1 -i_f_ ksh -c 'qrsh -cwd gzip -d _f_ &' \;
	### gzip -d ''filename''		### gzip -d ''filename''
	# make sure the files are readable by galaxy ~~user~~		# make sure the files are readable by galaxy
	## ~~Chmod og~~+r /lustre/scratch/''user''/proj1/*		## if you're in galaxy-admin UNIX group you can do
			### chgrp galaxy-admin *.fastq
			### chmod g+r *.fastq
			## if you're not, then you have to make it readable to the world (o=other)
			### chmod o+r /lustre/scratch/''user''/proj1/*

	== link into galaxy dataset (fast) ==		== link into galaxy dataset (fast) ==

Curtish@uab.edu at 20:56, 23 June 2011

2011-06-23T20:56:11Z

← Older revision		Revision as of 20:56, 23 June 2011
Line 10:		Line 10:
	### Chmod og+x /lustre/scratch/''user''/proj1		### Chmod og+x /lustre/scratch/''user''/proj1
	# transfer the files with SCP		# transfer the files with SCP
			## scp *.fastq.gz ''user''@cheaha.uabgrid.uab.edu:/lustre/scratch/''user''/proj1
	## I used "Secure Shell Client" for windows, available at [http://www.uab.edu/it/software/displaytitle.php?ItemID=34 UABIT]		## I used "Secure Shell Client" for windows, available at [http://www.uab.edu/it/software/displaytitle.php?ItemID=34 UABIT]
			## open source client is [http://www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY]
	# '''UNCOMPRESS fastq.gz files!!'''		# '''UNCOMPRESS fastq.gz files!!'''
	## cd /lustre/scratch/''user''/proj1		## cd /lustre/scratch/''user''/proj1
Line 27:		Line 29:
	### add Datasets		### add Datasets
	#### Upload option: Upload files from system path		#### Upload option: Upload files from system path
	##### ~~Using ssh run {~~find `pwd`~~} in~~ /~~lustrate~~/scratch/''user''/proj1/*		##### Get a list of absolute path names using one of the following
			###### cd /lustre/scratch/''user''/proj1 THEN RUN find `pwd` -name "*.fastq"
			###### find /lustre/scratch/''user''/proj1 -name "*.fastq"
	##### paste list of absolute path names into URL/Text box in Web Admin GUI		##### paste list of absolute path names into URL/Text box in Web Admin GUI
	#### Change "Copy data into Galaxy?" to "Link to files without copying into Galaxy"		#### Change "Copy data into Galaxy?" to "Link to files without copying into Galaxy"

Curtish@uab.edu at 22:02, 22 June 2011

2011-06-22T22:02:23Z

← Older revision		Revision as of 22:02, 22 June 2011
Line 11:		Line 11:
	# transfer the files with SCP		# transfer the files with SCP
	## I used "Secure Shell Client" for windows, available at [http://www.uab.edu/it/software/displaytitle.php?ItemID=34 UABIT]		## I used "Secure Shell Client" for windows, available at [http://www.uab.edu/it/software/displaytitle.php?ItemID=34 UABIT]
	#~~# UNCOMPRESSED~~ fastq.gz files!!		# '''UNCOMPRESS fastq.gz files!!'''
			## cd /lustre/scratch/''user''/proj1
			### find `pwd` -name "*.gz" -exec ksh -c 'qrsh "gzip -d \{}" &' \;
			### ls -1 *.gz \| xargs -L 1 -i_f_ ksh -c 'qrsh -cwd gzip -d _f_ &' \;
	### gzip -d ''filename''		### gzip -d ''filename''
	# make sure the files are readable by galaxy user		# make sure the files are readable by galaxy user

Curtish@uab.edu: Created page with "= Load and Link approach = == transfer and uncompress (slow) == # login to cheaha.uabgrid.uab.edu (linux), # create directory for this data set in your scratch dir ## mkdir /l..."

2011-06-22T20:49:21Z

Created page with "= Load and Link approach = == transfer and uncompress (slow) == # login to cheaha.uabgrid.uab.edu (linux), # create directory for this data set in your scratch dir ## mkdir /l..."

New page

= Load and Link approach =

== transfer and uncompress (slow) ==
# login to cheaha.uabgrid.uab.edu (linux),
# create directory for this data set in your scratch dir
## mkdir /lustre/scratch/''user''/proj1
## make sure that directory is readable by galaxy user
### Chmod og+x /lustre/scratch/''user''
### Chmod og+x /lustre/scratch/''user''/proj1
# transfer the files with SCP
## I used "Secure Shell Client" for windows, available at [http://www.uab.edu/it/software/displaytitle.php?ItemID=34 UABIT]
## UNCOMPRESSED fastq.gz files!!
### gzip -d ''filename''
# make sure the files are readable by galaxy user
## Chmod og+r /lustre/scratch/''user''/proj1/*

== link into galaxy dataset (fast) ==
# get Admin privileges on galaxy
## either get Shantanu to make you admin in Galaxy
## or grab someone who is (John, Curtis)
# In Galaxy GUI:
## admin > Manage Data Libraries > create new library
### add Datasets
#### Upload option: Upload files from system path
##### Using ssh run {find `pwd`} in /lustrate/scratch/''user''/proj1/*
##### paste list of absolute path names into URL/Text box in Web Admin GUI
#### Change "Copy data into Galaxy?" to "Link to files without copying into Galaxy"
#### Put something mnemonic in Message box.

== link data into a history (fast) ==

I could then select the datasets and, at bottom of page "For selected datasets: <Import to histories>" and get them into a history so I can compute on them.