Namespaces

Variants
Actions
Personal tools

Research Storage

From UABgrid Documentation

(Difference between revisions)
Jump to: navigation, search
(Description: Filled out with useful information about various storage locations)
(Added storage request info)
Line 39: Line 39:
  
 
===== `$SHARE_PROJECT` =====
 
===== `$SHARE_PROJECT` =====
Points to `/data/project/`. The preferred location for storing shared, research computing related files, scripts and code. To use this space you must make a formal request to [[Support]], please see that article for more details. Quotas are made on request for legitimate research projects or labs and the data in each quota must be owned by a particular PI on campus. The default quota size is 50 TB, but may be requested to be smaller. Larger quotas, or increasing an existing quota, requires justification.
+
Points to `/data/project/`. The preferred location for storing shared, research computing related files, scripts and code. To use this space you must make a formal request to [[Support]] using the information from the [[Support#Project_Requests|Project Request]] page. Quotas are made on request for legitimate research projects or labs and the data in each quota must be owned by a particular PI on campus. The default quota size is 50 TB, but may be requested to be smaller. Larger quotas, or increasing an existing quota, requires justification.
  
 
===== sloss =====
 
===== sloss =====
Line 45: Line 45:
  
 
== Request Storage ==
 
== Request Storage ==
 +
To request storage space, please contact [[Support]] using the information from the [[Support#Project_Requests|Project Request]] page.
 +
 
== How to Access ==
 
== How to Access ==
 
* From iMAC desktop running OS X, 10.9.5, use the following steps to mount Cheaha home directory for a BlazerID which has an establish account on Cheaha:
 
* From iMAC desktop running OS X, 10.9.5, use the following steps to mount Cheaha home directory for a BlazerID which has an establish account on Cheaha:

Revision as of 10:45, 30 July 2021

Research Storage is a scalable storage fabric designed to grow with your research operations. Data set sizes are growing dramatically. Keeping enough storage on hand to manage Big Data is a challenge for everyone. The Research Storage service provides nearly unlimited capacity to hold the data important to your research. The service is built using flexible technologies that can support any research data requirement.

Contents

Availability

A 1TB default allocation is available to individual users of the Cheaha HPC system to to help users adhere to cluster storage policies for their scratch and home directories and facilitate analyzing data on cluster. This storage space is available as an internal cluster file system via the standard SSH interface. Data can be transferred to and from the path `/rstore/user/$USER/default`.

IMPORTANT: Data on the cluster is not backed up by default. It is the responsibility of the user to ensure their research data sets are protected against loss. Please see other service offerings below if you are interested in backup services for your data.

Cluster scratch space is designated as a temporary storage location for actively analyzed data sets. Analysis results should be removed from scratch after jobs are completed. Cluster home directory space is designated for use as a workflow support space, holding custom codes and other tools needed to analyze data but not the data itself.

Because many data sets are large, transferring them to and from the cluster can require a significant amount of time. The research storage space now available on the cluster is designated as a place where users can keep data sets over longer periods of time for subsequent analysis and avoid crowding the scratch storage or abusing the home directory space.

Additional research storage space can be allocated for individuals or groups on request according to the rates for Research Storage.

Description

The following description of research storage locations and hardware assumes that all files and data will be placed on Cheaha with the intent of being used for computation related to a legitimate research need. Compute storage is not intended for archival or backup purposes, and is not intended to store personal, non-research related data. Research Computing does not own any data placed by users on Cheaha. Backup services are not provided by the Research Computing department and must be maintained by the data owner.

The following terms may be useful for the description of research storage:

  • GB - gigabyte
  • TB - terabyte (1024 GB)
  • PB - petabyte (1024 PB)
  • $ - anything following a $ symbol is a shell variable. All of the variables used here should be predefined in your shell environment on Cheaha
  • $USER - a shell variable containing your username/blazerid or XIAS account username


`$HOME`

Points to `/home/$USER`. Your home directory, where the operating system and most software stores personal configuration files. Historically on a separate filesystem from other storage locations, with a 20 GB quota. The previous hardware was retired and its data was placed on the primary `$USER_DATA` hardware and merged into that quota.

`$USER_DATA`

Points to `/data/user/$USER`. The preferred location for storing personal, research computing related files, scripts and code. Has a 5 TB quota, replicated for robustness.

`$USER_SCRATCH`

Points to `/scratch/$USER`. The preferred location for storing any temporary files related to research computing. Total space is 1 PB (petabyte) or 1024 TB shared among all users of the cluster. Users should delete files placed here on a regular basis or on job completion.

`$LOCAL_SCRATCH`

Points to `/scratch/local`. The preferred location for storing small quantities of temporary files for currently running jobs. Local to each compute node and not shared between nodes, but shared between all users. Typically this space is about 1 TB. Highest possible IO performance for a single node job. Users must delete files placed here on job completion.

`$SHARE_PROJECT`

Points to `/data/project/`. The preferred location for storing shared, research computing related files, scripts and code. To use this space you must make a formal request to Support using the information from the Project Request page. Quotas are made on request for legitimate research projects or labs and the data in each quota must be owned by a particular PI on campus. The default quota size is 50 TB, but may be requested to be smaller. Larger quotas, or increasing an existing quota, requires justification.

sloss

Not a shell variable, refers to `/data/project/sloss`. A special project location under `$SHARE_PROJECT` for projects that are at most a few TB. Essentially a foundry for project spaces that start small but may grow and graduate into a full-fledged project space.

Request Storage

To request storage space, please contact Support using the information from the Project Request page.

How to Access

  • From iMAC desktop running OS X, 10.9.5, use the following steps to mount Cheaha home directory for a BlazerID which has an establish account on Cheaha:
    • Click on "Go" from menu bar at top of screen;
    • New window "Connect to Server" opens;
    • There are two screens within "Connect to Server" window, which are: "Server Address:” and “Favorite Servers:”
    • For example, some possible server addresses within Favorite Servers:
      • smb://uabfile.ad.uab.edu/ShareNAME, where ShareNAME is the name of a share on UABFILE;
      • smb://138.26.XYZ.ABC, where XYZ.ABC is the complete IP address of a server within the UAB domain 138.68...
      • smb://files.uabgrid.uab.edu
    • one can enter a new command in the "connect to Server" window.