ResearchStorage - Revision history

Jpr@uab.edu: /* How to get more */ Changed the wording to not imply we are tracking the use time period.

2014-04-29T16:38:31Z

How to get more: Changed the wording to not imply we are tracking the use time period.

← Older revision		Revision as of 16:38, 29 April 2014
Line 37:		Line 37:
	== How to get more ==		== How to get more ==

	You can buy any amount of additional storage at a rate of $0.38/Gigabyte/year. That's $395/Terabyte/year. All we need to know is how much storage you want~~, for how long,~~ and a billing account number. UAB IT will bill you monthly for the storage you consume.		You can buy any amount of additional storage at a rate of $0.38/Gigabyte/year. That's $395/Terabyte/year. All we need to know is how much storage you want and a billing account number. UAB IT will bill you monthly for the storage you consume. When you are done using the storage, let us know, we will delete the storage and stop billing you for it.

	== What about backups? ==		== What about backups? ==

Jpr@uab.edu: Fixed typos and tuned some wording.

2014-04-16T23:42:42Z

Fixed typos and tuned some wording.

Jpr@uab.edu: /* No Backups! */ Fix typos and slang

2014-04-16T20:42:22Z

No Backups!: Fix typos and slang

← Older revision		Revision as of 20:42, 16 April 2014
Line 46:		Line 46:
	''This rule includes the new Research Storage containers.''		''This rule includes the new Research Storage containers.''

	Please understand we do not say this out of malice or a lack of concern for your data. Central backup processes are inherently ignorant and must assume all files are important. This is done at the very real expense of keeping multiple copies of data. In the context of large scale data sets which are typical of our HPC environment, this would amount to 100's of terabytes of data. Duplicating the footprint of data for which we are already ~~busting~~ at the seams to support a single copy.		Please understand we do not say this out of malice or a lack of concern for your data. Central backup processes are inherently ignorant and must assume all files are important. This is done at the very real expense of keeping multiple copies of data. In the context of large scale data sets which are typical of our HPC environment, this would amount to 100's of terabytes of data. Duplicating the footprint of data for which we are already bursting at the seams to support a single copy.

	It is much better for individuals, teams, or labs and their technical support staff critical data and ensure it is backed up.		It is much better for individuals, teams, or labs and their technical support staff to identify critical data and ensure it is backed up.

	=== How to Backup ===		=== How to Backup ===

Jpr@uab.edu: Add new section on what the user gets.

2014-04-16T20:40:16Z

Add new section on what the user gets.

← Older revision		Revision as of 20:40, 16 April 2014
Line 1:		Line 1:
	Research Storage provides versatile containers for your data sets. The containers include dedicated space available on the Cheaha HPC platform.		Research Storage provides versatile containers for your data sets. The containers include dedicated space available on the Cheaha HPC platform.

			== What do I get ==

			Each active user on Cheaha receives an annual 1-terabyte allocation of Research Storage at no direct cost to the user. This container is dedicated to your storage needs and is attached directly to the cluster.

			Additional storage capacity can purchased, see details below.

	== How to get it ==		== How to get it ==

Jpr@uab.edu: /* How to get it */ Add human readable flag to df

2014-04-16T20:24:16Z

How to get it: Add human readable flag to df

← Older revision		Revision as of 20:24, 16 April 2014
Line 9:		Line 9:
	You can check to see how much storage you have used in your container with the command:		You can check to see how much storage you have used in your container with the command:

	df /rstore/user/$USER/default		df -h /rstore/user/$USER/default

	== How to use it ==		== How to use it ==

Jpr@uab.edu: /* How to use it */ Add subsections and improve formatting

2014-04-16T20:22:09Z

How to use it: Add subsections and improve formatting

← Older revision		Revision as of 20:22, 16 April 2014
Line 13:		Line 13:
	== How to use it ==		== How to use it ==

			=== For HPC work flows ===
	You can use this storage in any way that you find useful to assist you with your HPC work flows on Cheaha.		You can use this storage in any way that you find useful to assist you with your HPC work flows on Cheaha.

	~~But, you~~ should still follow good high performance compute work flow recommendations and stage data sets $USER_SCRATCH during your job runs, especially if those data sets are heavily accessed or modified as part of your job operations.		You should still follow good high performance compute work flow recommendations and stage data sets $USER_SCRATCH during your job runs, especially if those data sets are heavily accessed or modified as part of your job operations.

	~~One near term~~ use ~~would be to safely preserve important~~ data in your ~~$USER_SCRATCH so~~ that it is ~~not destroyed by the upcoming scratch file system rebuild during the May 3-10 cluster service window~~.		In general, a good use this storage for keeping larger data sets on the cluster longer than the lifetime of your active computations and for stuff that is too big to fit in your home directory.

			=== For Retaining Scratch Data ===

	~~Follow these steps:~~		One near term use for your storage container would be to safely preserve important data in your $USER_SCRATCH so that it is not destroyed by the upcoming scratch file system rebuild during the May 3-10 cluster service window.

			You can follow these steps to move a copy of important files from your personal scratch space to your default Research Storage container:
	# Remove any data from $USER_SCRATCH that you no longer use or want		# Remove any data from $USER_SCRATCH that you no longer use or want
	# Copy your remaining important data to your default container:		# Copy your remaining important data to your default container using rsync
	rsync -a --stats $USER_SCRATCH/ /rstore/user/$USER/default/scratch		rsync -a --stats $USER_SCRATCH/ /rstore/user/$USER/default/scratch

	~~In general, a good use this storage for keeping larger data sets on the cluster longer than the lifetime of your active computations and for stuff that is too big to fit in your home directory.~~

	== How to get more ==		== How to get more ==

Jpr@uab.edu: /* What about backups? */ Improve the wording and formatting to highlight policy

2014-04-16T20:09:47Z

What about backups?: Improve the wording and formatting to highlight policy

← Older revision		Revision as of 20:09, 16 April 2014
Line 33:		Line 33:
	== What about backups? ==		== What about backups? ==

	There is no central back up process on the cluster. ~~This rule includes~~ the ~~new Research Storage containers~~.		=== No Backups! ===
			There is no central back up process on the cluster. Each user is responsible for backing up their own data. If you are not managing a back up process for data on the cluster then you do not have any back ups.

	~~Each user is responsible for backing up their own data!~~		''This rule includes the new Research Storage containers.''

	~~If you are not managing a back up process for data on~~ the ~~cluster then you do not have any back ups~~.

	Please understand we do not say this out of malice or a lack of concern for your data. Central backup processes are inherently ignorant and must assume all files are important. This is done at the very real expense of keeping multiple copies of data. In the context of large scale data sets which are typical of our HPC environment, this would amount to 100's of terabytes of data. Duplicating the footprint of data for which we are already busting at the seams to support a single copy.		Please understand we do not say this out of malice or a lack of concern for your data. Central backup processes are inherently ignorant and must assume all files are important. This is done at the very real expense of keeping multiple copies of data. In the context of large scale data sets which are typical of our HPC environment, this would amount to 100's of terabytes of data. Duplicating the footprint of data for which we are already busting at the seams to support a single copy.
Line 43:		Line 42:
	It is much better for individuals, teams, or labs and their technical support staff critical data and ensure it is backed up.		It is much better for individuals, teams, or labs and their technical support staff critical data and ensure it is backed up.

			=== How to Backup ===
	We understand this process can be difficult, especially if you are your own technical support staff.		We understand this process can be difficult, especially if you are your own technical support staff.

	To that end, we have a new backup service available that leverages CrashPlan, a popular commercial backup product that will help you easily back up your data on your laptop or in your lab.		To that end, we have a new backup service available that leverages CrashPlan, a popular commercial backup product that will help you easily back up your data on your laptop or in your lab.

	Please contact us if you are interested in using CrashPlan to ~~fulfill~~ your responsibilities for backing up your data.		Please contact us if you are interested in using CrashPlan to fulfil your responsibilities for backing up your own data.

Jpr@uab.edu: /* How to use it */ fix bullet formatting

2014-04-16T20:01:49Z

How to use it: fix bullet formatting

← Older revision		Revision as of 20:01, 16 April 2014
Line 21:		Line 21:
	Follow these steps:		Follow these steps:

	Remove any data from $USER_SCRATCH that you no longer use or want		# Remove any data from $USER_SCRATCH that you no longer use or want
	Copy your remaining important data to your default container:		# Copy your remaining important data to your default container:
	rsync -a --stats $USER_SCRATCH/ /rstore/user/$USER/default/scratch		rsync -a --stats $USER_SCRATCH/ /rstore/user/$USER/default/scratch


	In general, a good use this storage for keeping larger data sets on the cluster longer than the lifetime of your active computations and for stuff that is too big to fit in your home directory.		In general, a good use this storage for keeping larger data sets on the cluster longer than the lifetime of your active computations and for stuff that is too big to fit in your home directory.

Jpr@uab.edu: Got rid of the boring text and replaced it with more relavent text from email

2014-04-16T20:00:35Z

Got rid of the boring text and replaced it with more relavent text from email

← Older revision		Revision as of 20:00, 16 April 2014
Line 1:		Line 1:
	Research Storage provides versatile containers for your data sets. The containers include dedicated space available on the Cheaha HPC platform ~~and room for maintaining back-ups. Their flexibility allows you to easily capacity expansion. they can support the construction of new services and applicatons into the future~~.		Research Storage provides versatile containers for your data sets. The containers include dedicated space available on the Cheaha HPC platform.

	~~The Research Storage service was developed by UAB IT's Research Computing group~~ to ~~support the ever increasing demand for space in modern era of data intensive science.~~		== How to get it ==

	~~== Availability ==~~		Once you log in to [[Cheaha]], you can access your default 1TB storage container here:

	~~As part of the initial release, Research Storage is only accessible through the Cheaha HPC platform. Cluster users can access an automatically provisioned "default" container at the path~~ /rstore/user/$USER/default.		cd /rstore/user/$USER/default

	~~== Cost ==~~		You can check to see how much storage you have used in your container with the command:

	~~Research storage is charged per gigabyte used at a rate of~~ $~~0.39 per Gigabyte per year. This is an annual cost of $395 per Terabyte.~~		df /rstore/user/$USER/default

	==~~= First Terabyte Free =~~==		== How to use it ==

	~~Researchers that~~ use ~~Cheaha receive an annual allocation of 1TB of~~ storage with ~~their account~~. ~~This storage containers is known as their "default" container and is accessible at the path /rstore/user/$USER/default:~~		You can use this storage in any way that you find useful to assist you with your HPC work flows on Cheaha.
	~~cd /rstore/user/$USER/default~~

	~~Additional containers can~~ be ~~provisioned according~~ to the amount of storage ~~needed~~.		But, you should still follow good high performance compute work flow recommendations and stage data sets $USER_SCRATCH during your job runs, especially if those data sets are heavily accessed or modified as part of your job operations.

			One near term use would be to safely preserve important data in your $USER_SCRATCH so that it is not destroyed by the upcoming scratch file system rebuild during the May 3-10 cluster service window.

			Follow these steps:

			Remove any data from $USER_SCRATCH that you no longer use or want
			Copy your remaining important data to your default container:
			rsync -a --stats $USER_SCRATCH/ /rstore/user/$USER/default/scratch


			In general, a good use this storage for keeping larger data sets on the cluster longer than the lifetime of your active computations and for stuff that is too big to fit in your home directory.

			== How to get more ==

			You can buy any amount of additional storage at a rate of $0.38/Gigabyte/year. That's $395/Terabyte/year. All we need to know is how much storage you want, for how long, and an account number. UAB IT will bill you monthly for the storage you consume

			== What about backups? ==

			There is no central back up process on the cluster. This rule includes the new Research Storage containers.

			Each user is responsible for backing up their own data!

			If you are not managing a back up process for data on the cluster then you do not have any back ups.

			Please understand we do not say this out of malice or a lack of concern for your data. Central backup processes are inherently ignorant and must assume all files are important. This is done at the very real expense of keeping multiple copies of data. In the context of large scale data sets which are typical of our HPC environment, this would amount to 100's of terabytes of data. Duplicating the footprint of data for which we are already busting at the seams to support a single copy.

			It is much better for individuals, teams, or labs and their technical support staff critical data and ensure it is backed up.

			We understand this process can be difficult, especially if you are your own technical support staff.

			To that end, we have a new backup service available that leverages CrashPlan, a popular commercial backup product that will help you easily back up your data on your laptop or in your lab.

			Please contact us if you are interested in using CrashPlan to fulfill your responsibilities for backing up your data.

Jpr@uab.edu at 18:42, 16 April 2014

2014-04-16T18:42:12Z

← Older revision		Revision as of 18:42, 16 April 2014
Line 1:		Line 1:
	~~Research Storage~~

	Research Storage provides versatile containers for your data sets. The containers include dedicated space available on the Cheaha HPC platform and room for maintaining back-ups. Their flexibility allows you to easily capacity expansion. they can support the construction of new services and applicatons into the future.		Research Storage provides versatile containers for your data sets. The containers include dedicated space available on the Cheaha HPC platform and room for maintaining back-ups. Their flexibility allows you to easily capacity expansion. they can support the construction of new services and applicatons into the future.

← Older revision		Revision as of 23:42, 16 April 2014
Line 5:		Line 5:
	Each active user on Cheaha receives an annual 1-terabyte allocation of Research Storage at no direct cost to the user. This container is dedicated to your storage needs and is attached directly to the cluster.		Each active user on Cheaha receives an annual 1-terabyte allocation of Research Storage at no direct cost to the user. This container is dedicated to your storage needs and is attached directly to the cluster.

	Additional storage capacity can purchased, see details below.		Additional storage capacity can be purchased, see details below.

	== How to get it ==		== How to get it ==
Line 22:		Line 22:
	You can use this storage in any way that you find useful to assist you with your HPC work flows on Cheaha.		You can use this storage in any way that you find useful to assist you with your HPC work flows on Cheaha.

	You should still follow good high performance compute work flow recommendations and stage data sets $USER_SCRATCH during your job runs, especially if those data sets are heavily accessed or modified as part of your job operations.		You should still follow good high performance compute work flow recommendations and stage data sets in your $USER_SCRATCH during your job runs, especially if those data sets are heavily accessed or modified as part of your job operations.

	In general, a good use this storage for keeping larger data sets on the cluster longer than the lifetime of your active computations and for stuff that is too big to fit in your home directory.		In general, a good use of this storage is for keeping larger data sets on the cluster longer than the lifetime of your active computations and for stuff that is too big to fit in your home directory.

	=== For Retaining Scratch Data ===		=== For Retaining Scratch Data ===

	One near term use for your storage container would be to safely preserve important data in your $USER_SCRATCH so that it is not destroyed by the upcoming scratch file system rebuild during the May 3-10 cluster service window.		One near term use for your storage container would be to safely preserve important data in your $USER_SCRATCH so that it is not destroyed by the upcoming scratch file system rebuild. These destructive data operations are scheduled during the May 3-10 2014 cluster service window.

	~~You can follow~~ these steps to move a copy of important files from your personal scratch space to your default Research Storage container:		Follow these steps to move a copy of important files from your personal scratch space to your default Research Storage container:
	# Remove any data from $USER_SCRATCH that you no longer use or want		# Remove any data from $USER_SCRATCH that you no longer use or want
	# Copy your remaining important data to your default container using rsync		# Copy your remaining important data to your default Research Storage container using rsync
	rsync -a --stats $USER_SCRATCH/ /rstore/user/$USER/default/scratch		rsync -a --stats $USER_SCRATCH/ /rstore/user/$USER/default/scratch

	== How to get more ==		== How to get more ==

	You can buy any amount of additional storage at a rate of $0.38/Gigabyte/year. That's $395/Terabyte/year. All we need to know is how much storage you want, for how long, and an account number. UAB IT will bill you monthly for the storage you consume		You can buy any amount of additional storage at a rate of $0.38/Gigabyte/year. That's $395/Terabyte/year. All we need to know is how much storage you want, for how long, and a billing account number. UAB IT will bill you monthly for the storage you consume.

	== What about backups? ==		== What about backups? ==

	=== No Backups! ===		=== No Backups! ===
	There is no central back up process on the cluster. Each user is responsible for backing up their own data. If you are not managing a back up process for data on the cluster then you do not have any ~~back ups~~.		There is no central back up process on the cluster. Each user is responsible for backing up their own data. If you are not managing a back up process for your data on the cluster then you do not have any backups.

	''This rule ~~includes~~ the new Research Storage containers.''		''This rule also applies to the new Research Storage containers.''

	Please understand we do not say this out of malice or a lack of concern for your data. Central backup processes are inherently ignorant and must assume all files are important. This is done at the very real expense of keeping multiple copies of data. In the context of large scale data sets which are typical of our HPC environment, this would amount to 100's of terabytes of data. ~~Duplicating~~ the footprint of data for which we are already bursting at the seams to support a single copy.		Please understand we do not say this out of malice or a lack of concern for your data. Central backup processes are inherently ignorant and must assume all files are important. This is done at the very real expense of keeping multiple copies of data. In the context of large scale data sets, which are typical in our HPC environment, this would amount to storing 100's of terabytes of data. It would duplicating the footprint of data for which we are already bursting at the seams in order to support a single copy.

	It is much better for individuals, teams, or labs and their technical support staff to identify critical data and ensure it is backed up.		It is much better for individuals, teams, or labs and their technical support staff to identify critical data and ensure it is backed up.
Line 55:		Line 55:
	To that end, we have a new backup service available that leverages CrashPlan, a popular commercial backup product that will help you easily back up your data on your laptop or in your lab.		To that end, we have a new backup service available that leverages CrashPlan, a popular commercial backup product that will help you easily back up your data on your laptop or in your lab.

	Please contact us if you are interested in using CrashPlan to fulfil your responsibilities ~~for~~ backing up your own data.		Please contact us if you are interested in using CrashPlan to fulfil your responsibilities toward backing up your own data.