UploadLargeData: Difference between revisions

From Cheaha
Jump to navigation Jump to search
(Added image)
(removed page. Left links to other related pages. It was redundent.)
 
Line 1: Line 1:
= Load and Link approach =
= Page removed =


 
Please see [[Galaxy_File_Uploads]] for an overview and [[Galaxy_Data_Import]] for large datasets.
== transfer and uncompress (slow) ==
# login to cheaha.uabgrid.uab.edu (linux),
# create directory for this data set in your scratch dir
## mkdir /lustre/scratch/''user''/proj1
## make sure that directory is readable by galaxy user
### Chmod og+x /lustre/scratch/''user''
### Chmod og+x /lustre/scratch/''user''/proj1
# transfer the files with SCP
## scp *.fastq.gz ''user''@cheaha.uabgrid.uab.edu:/lustre/scratch/''user''/proj1
## I used "Secure Shell Client" for windows, available at [http://www.uab.edu/it/software/displaytitle.php?ItemID=34 UABIT]
## open source client is [http://www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY]
# '''UNCOMPRESS fastq.gz files!!'''
## cd /lustre/scratch/''user''/proj1
### gzip -d ''filename''
### !!WARNING: the following parallel decompress stesp fail for docs over 8G uncompressed!!
### find `pwd` -name "*.gz" -exec ksh -c 'qrsh "gzip -d \{}" &' \;
### ls -1 *.gz | xargs -L 1 -i_f_ ksh -c 'qrsh -cwd gzip -d _f_ &' \;
# make sure the files are readable by galaxy
## if you're in galaxy-admin UNIX group you can do
### chgrp galaxy-admin *.fastq
### chmod g+r *.fastq
## if you're not, then you have to make it readable to the world (o=other)
###  chmod o+r /lustre/scratch/''user''/proj1/*
 
== link into galaxy dataset (fast) ==
# get Admin privileges on galaxy
## either get Shantanu to make you admin in Galaxy
## or grab someone who is (John, Curtis)
# In Galaxy GUI:
## admin > Manage Data Libraries > create new library
### add Datasets
#### Upload option: Upload files from system path
##### Get a list of absolute path names using one of the following
###### cd /lustre/scratch/''user''/proj1 THEN RUN find `pwd` -name "*.fastq"
###### find /lustre/scratch/''user''/proj1 -name "*.fastq"
##### paste list of absolute path names into URL/Text box in Web Admin GUI
#### Change "Copy data into Galaxy?" to "Link to files without copying into Galaxy"
#### Put something mnemonic in Message box.
[[File:UploadLinkDataFilesGUI.jpg]]
 
== link data into a history (fast) ==
 
I could then select the datasets and, at bottom of page "For selected datasets: <Import to histories>" and get them into a history so I can compute on them.

Latest revision as of 19:37, 7 March 2013

Page removed

Please see Galaxy_File_Uploads for an overview and Galaxy_Data_Import for large datasets.

Retrieved from "https://docs.uabgrid.uab.edu/w/index.php?title=UploadLargeData&oldid=4404"