Data Movement - Revision history

Louistw@uab.edu: update rclone setting video

2021-04-27T15:49:02Z

update rclone setting video

← Older revision		Revision as of 15:49, 27 April 2021
Line 12:		Line 12:

	== Rclone ==		== Rclone ==
			Here's [https://www.youtube.com/watch?v=UbFJV9TO4KE example video] to setup rclone for [https://box.com box.com].

	Data Transfer Cheaha to BOX. In the terminal (inside the VNC session), load the module rclone		Data Transfer Cheaha to BOX. In the terminal (inside the VNC session), load the module rclone

Jpr@uab.edu: Add further reading section with link to uci howto

2020-10-15T17:23:47Z

Add further reading section with link to uci howto

← Older revision		Revision as of 17:23, 15 October 2020
Line 198:		Line 198:

	Downloading files, or complete directories, works essentially the same way as uploading - you just drag the files/directories from the remote pane to the local pane this time, instead of the other way round.		Downloading files, or complete directories, works essentially the same way as uploading - you just drag the files/directories from the remote pane to the local pane this time, instead of the other way round.

			== Further Reading ==

			A good primer on data movement for researchers is [http://moo.nac.uci.edu/~hjm/HOWTO_move_data.html Harry Mangalam's HOW-TO on moving data].

Jpr@uab.edu: Add Globus section and link to page.

2020-10-15T17:13:21Z

Add Globus section and link to page.

← Older revision		Revision as of 17:13, 15 October 2020
Line 3:		Line 3:
	There are various Linux native commands that you can use to move your data within the HPC cluster, such as [https://linux.die.net/man/1/mv mv], [https://linux.die.net/man/1/cp cp], [https://linux.die.net/man/1/scp scp] etc. One of the most powerful tools for data movement on Linux is [https://linux.die.net/man/1/rsync rsync], which we'll be using in our examples below.		There are various Linux native commands that you can use to move your data within the HPC cluster, such as [https://linux.die.net/man/1/mv mv], [https://linux.die.net/man/1/cp cp], [https://linux.die.net/man/1/scp scp] etc. One of the most powerful tools for data movement on Linux is [https://linux.die.net/man/1/rsync rsync], which we'll be using in our examples below.

	'''rsync''' and '''scp''' can also be used for moving data from a local storage to Cheaha.		'''rsync''' and '''scp''' can also be used for moving data from a local storage to Cheaha.

			For moving large volumes of data to and from the cluster we also recommend the [[Globus]] service.

			== Globus ==

			Please see the dedicated page on moving data with [[Globus]]. This is a good option if the site or user you are collaborating has Globus but may require the installation of a transfer agent on the users computer.

	== Rclone ==		== Rclone ==

Krish94@uab.edu: /* Rclone */

2019-10-09T17:37:49Z

Rclone

← Older revision		Revision as of 17:37, 9 October 2019
Line 52:		Line 52:
	y/e/d> y		y/e/d> y

			rclone ls
			List the objects in the path with size and path.
			rclone ls remote:path [flags]

			rclone copy
			Copy files from source to dest, skipping already copied.
			Note: Use the -P/--progress flag to view real-time transfer statistics
			rclone copy source:sourcepath dest:destpath

	==RSync==		==RSync==

Krish94@uab.edu: added rclone instructions to transfer data from cheaha to box

2019-09-10T20:07:35Z

added rclone instructions to transfer data from cheaha to box

← Older revision		Revision as of 20:07, 10 September 2019
Line 4:		Line 4:

	'''rsync''' and '''scp''' can also be used for moving data from a local storage to Cheaha.		'''rsync''' and '''scp''' can also be used for moving data from a local storage to Cheaha.

			== Rclone ==

			Data Transfer Cheaha to BOX. In the terminal (inside the VNC session), load the module rclone
			module load rclone/1.48.0
			The initial setup for Box involves getting a token from Box rclone config walks you through it.
			Here is an example of how to make a remote called remote. First run:
			rclone config
			This will guide you through an interactive setup process:
			No remotes found - make a new one
			n) New remote
			s) Set configuration password
			q) Quit config
			n/s/q> n
			name> remote
			Type of storage to configure.
			Choose a number from below, or type in your own value
			[snip]
			XX / Box
			\ "box"
			[snip]
			Storage> box
			Box App Client Id - leave blank normally.
			client_id>
			Box App Client Secret - leave blank normally.
			client_secret>
			Remote config
			Use auto config?
			* Say Y if not sure
			* Say N if you are working on a remote or headless machine
			y) Yes
			n) No
			y/n> y
			If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth
			Log in and authorize rclone for access
			Waiting for code...
			Got code
			--------------------
			[remote]
			client_id =
			client_secret =
			token = {"access_token":"XXX","token_type":"bearer","refresh_token":"XXX","expiry":"XXX"}
			--------------------
			y) Yes this is OK
			e) Edit this remote
			d) Delete this remote
			y/e/d> y


	==RSync==		==RSync==

Ravi89@uab.edu at 15:51, 15 August 2019

2019-08-15T15:51:55Z

← Older revision		Revision as of 15:51, 15 August 2019
Line 1:		Line 1:
	'''NOTE: If you find better and faster methods/tools, please' add them to this page as well''		'''NOTE: If you find better and faster methods/tools, please add them to this page as well''

	There are various Linux native commands that you can use to move your data within the HPC cluster, such as [https://linux.die.net/man/1/mv mv], [https://linux.die.net/man/1/cp cp], [https://linux.die.net/man/1/scp scp] etc. One of the most powerful tools for data movement on Linux is [https://linux.die.net/man/1/rsync rsync], which we'll be using in our examples below.		There are various Linux native commands that you can use to move your data within the HPC cluster, such as [https://linux.die.net/man/1/mv mv], [https://linux.die.net/man/1/cp cp], [https://linux.die.net/man/1/scp scp] etc. One of the most powerful tools for data movement on Linux is [https://linux.die.net/man/1/rsync rsync], which we'll be using in our examples below.

Ravi89@uab.edu at 15:51, 15 August 2019

2019-08-15T15:51:32Z

← Older revision		Revision as of 15:51, 15 August 2019
Line 1:		Line 1:
	'''NOTE: ~~This~~ page ~~is under construction.'~~''		'''NOTE: If you find better and faster methods/tools, please' add them to this page as well''

	There are various Linux native commands that you can use to move your data within the HPC cluster, such as [https://linux.die.net/man/1/mv mv], [https://linux.die.net/man/1/cp cp], [https://linux.die.net/man/1/scp scp] etc. One of the most powerful tools for data movement on Linux is [https://linux.die.net/man/1/rsync rsync], which we'll be using in our examples below.		There are various Linux native commands that you can use to move your data within the HPC cluster, such as [https://linux.die.net/man/1/mv mv], [https://linux.die.net/man/1/cp cp], [https://linux.die.net/man/1/scp scp] etc. One of the most powerful tools for data movement on Linux is [https://linux.die.net/man/1/rsync rsync], which we'll be using in our examples below.

Ravi89@uab.edu: /* Jobs */

2018-06-04T23:26:09Z

Jobs

← Older revision		Revision as of 23:26, 4 June 2018
Line 58:		Line 58:
	{{SensitiveInformation}}		{{SensitiveInformation}}

	==~~Jobs~~==		==Moving large amount of data on cheaha==

	If the data that you are moving is large, then you should always use either an interactive session or a job script for your data movement. This ensures that the process for your data movement isn't using and slowing login nodes for a long time, and instead is performing these operations on a compute node.		If the data that you are moving is large, then you should always use either an interactive session or a job script for your data movement. This ensures that the process for your data movement isn't using and slowing login nodes for a long time, and instead is performing these operations on a compute node.

Ravi89@uab.edu: /* RSync */ Including instructions to move data to/from cheaha using rsync

2018-06-04T23:24:26Z

RSync: Including instructions to move data to/from cheaha using rsync

← Older revision		Revision as of 23:24, 4 June 2018
Line 36:		Line 36:

	If you are interested in finding out about various methods of moving data and various tools which can be used to achieve that aim, this page provides a very good description/guide : [http://moo.nac.uci.edu/~hjm/HOWTO_move_data.html How to transfer large amounts of data via network.].		If you are interested in finding out about various methods of moving data and various tools which can be used to achieve that aim, this page provides a very good description/guide : [http://moo.nac.uci.edu/~hjm/HOWTO_move_data.html How to transfer large amounts of data via network.].

			===Moving local data to Cheaha===
			To move data from your local storage to cheaha, using rsync, you can use the following command, on your local system:
			<pre>
			rsync -aP PATH_TO_FILE_OR_DIRECTORY BLAZERID@cheaha.rc.uab.edu:PATH_ON_CHEAHA
			</pre>
			where,
			* PATH_TO_FILE_OR_DIRECTORY is the full/relative path to file or directory that you want to move to cheaha. If transferring a directory, a trailing slash on the source creates an additional directory level at the destination. You can think of a trailing / on a source as meaning "copy the contents of this directory" as opposed to "copy the directory by name"
			* PATH_ON_CHEAHA is the full path on cheaha, where you want to move your directory. Remember, you have 20GB of space in your HOME directory, os if you are moving raw data to cheaha, please move it in either, /data/user/$BLAZERID or /data/scratch/$BLAZERID

			===Moving data on Cheaha to your local storage===
			To move data from cheaha to your local storage, using rsync, you can use the following command, on your local system:
			<pre>
			rsync -aP BLAZERID@cheaha.rc.uab.edu:PATH_TO_FILE_OR_DIRECTORY_ON_CHEAHA PATH_ON_LOCAL_STORAGE
			</pre>
			where,
			* PATH_TO_FILE_OR_DIRECTORY_ON_CHEAHA is the full path to file or directory on cheaha that you want to move to your local storage. If transferring a directory, a trailing slash on the source creates an additional directory level at the destination. You can think of a trailing / on a source as meaning "copy the contents of this directory" as opposed to "copy the directory by name"
			* PATH_ON_LOCAL_STORAGE is the full/relative path on cheaha, where you want to move your directory.

	== Privacy ==		== Privacy ==

Ravi89@uab.edu at 17:40, 22 May 2018

2018-05-22T17:40:52Z

← Older revision		Revision as of 17:40, 22 May 2018
Line 81:		Line 81:
	* Please change the time required and the corresponding [https://docs.uabgrid.uab.edu/wiki/SLURM#Slurm_Partitions partition] according to your need.		* Please change the time required and the corresponding [https://docs.uabgrid.uab.edu/wiki/SLURM#Slurm_Partitions partition] according to your need.
	* After modifications to the given job script, submit it using : '''sbatch JOB_SCRIPT'''		* After modifications to the given job script, submit it using : '''sbatch JOB_SCRIPT'''

			==FileZilla==

	'''Installation		'''Installation