UploadLargeData

From UABgrid Documentation
(Difference between revisions)
Jump to: navigation, search
(Added image)
(removed page. Left links to other related pages. It was redundent.)
 
Line 1: Line 1:
= Load and Link approach =
+
= Page removed =
  
 
+
Please see [[Galaxy_File_Uploads]] for an overview and [[Galaxy_Data_Import]] for large datasets.
== transfer and uncompress (slow) ==
+
# login to cheaha.uabgrid.uab.edu (linux),
+
# create directory for this data set in your scratch dir
+
## mkdir /lustre/scratch/''user''/proj1
+
## make sure that directory is readable by galaxy user
+
### Chmod og+x /lustre/scratch/''user''
+
### Chmod og+x /lustre/scratch/''user''/proj1
+
# transfer the files with SCP
+
## scp *.fastq.gz ''user''@cheaha.uabgrid.uab.edu:/lustre/scratch/''user''/proj1
+
## I used "Secure Shell Client" for windows, available at [http://www.uab.edu/it/software/displaytitle.php?ItemID=34 UABIT]
+
## open source client is [http://www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY]
+
# '''UNCOMPRESS fastq.gz files!!'''
+
## cd /lustre/scratch/''user''/proj1
+
### gzip -d ''filename''
+
### !!WARNING: the following parallel decompress stesp fail for docs over 8G uncompressed!!
+
### find `pwd` -name "*.gz" -exec ksh -c 'qrsh "gzip -d \{}" &' \;
+
### ls -1 *.gz | xargs -L 1 -i_f_ ksh -c 'qrsh -cwd gzip -d _f_ &' \;
+
# make sure the files are readable by galaxy
+
## if you're in galaxy-admin UNIX group you can do
+
### chgrp galaxy-admin *.fastq
+
### chmod g+r *.fastq
+
## if you're not, then you have to make it readable to the world (o=other)
+
###  chmod o+r /lustre/scratch/''user''/proj1/*
+
 
+
== link into galaxy dataset (fast) ==
+
# get Admin privileges on galaxy
+
## either get Shantanu to make you admin in Galaxy
+
## or grab someone who is (John, Curtis)
+
# In Galaxy GUI:
+
## admin > Manage Data Libraries > create new library
+
### add Datasets
+
#### Upload option: Upload files from system path
+
##### Get a list of absolute path names using one of the following
+
###### cd /lustre/scratch/''user''/proj1 THEN RUN find `pwd` -name "*.fastq"
+
###### find /lustre/scratch/''user''/proj1 -name "*.fastq"
+
##### paste list of absolute path names into URL/Text box in Web Admin GUI
+
#### Change "Copy data into Galaxy?" to "Link to files without copying into Galaxy"
+
#### Put something mnemonic in Message box.
+
[[File:UploadLinkDataFilesGUI.jpg]]
+
 
+
== link data into a history (fast) ==
+
 
+
I could then select the datasets and, at bottom of page "For selected datasets: <Import to histories>" and get them into a history so I can compute on them.
+

Latest revision as of 13:37, 7 March 2013

[edit] Page removed

Please see Galaxy_File_Uploads for an overview and Galaxy_Data_Import for large datasets.

Personal tools
Namespaces

Variants
Actions
Navigation
Toolbox