https://docs.uabgrid.uab.edu/w/index.php?action=history&feed=atom&title=Data_Management_Framework 2026-05-10T05:12:10Z Revision history for this page on the wiki MediaWiki 1.38.2 https://docs.uabgrid.uab.edu/w/index.php?title=Data_Management_Framework&diff=3517&oldid=prev 2011-12-12T17:40:59Z

<p>Create draft of framework from email thread</p> <p><b>New page</b></p><div>The Research Computing Platform supports defining and curating data sets for use by researchers directly or by reference in data analysis packages.<br /> <br /> == Philosophy of Data Management Framework ==<br /> <br /> Data sets should be treated in a manner akin to apps. Different apps can have different admin/owner groups, organized by app. An app is a work product whose outcome is a curated application install. These apps go in /share/apps/&lt;apptag&gt; and the permissions are defined based on the group maintaining the app.<br /> <br /> Similarly, data sets should be considered as work products whose outcome is a curated data set. As with applications, there is no single group that will manage all data sets. Data sets should be organized in /luster/projects/public-datasets/&lt;datasettag&gt; (or better /lustre/data/&lt;datasettag&gt;). Permissions on /lustre/data/&lt;datasettag&gt; should be based on people who are agreeing to maintain a specific &lt;datasettag&gt;. Some users will be admins on multiple data sets; some groups may bundle a bunch of data sets under one datasettag, others may prefer a strict separations dictated by upstream sources or orgs. (Think github here.)<br /> <br /> == Galaxy Example ==<br /> <br /> Considering the [[Galaxy]] application, the current /lustre/project/galaxy/public-datasets fits into the above model if you think of this as a curated data set for Galaxy where the dataset admins have chosen to treat a number of distinct data sets as part of a single collection. This also facilitates developing datasets with additional artifacts that support inclusion in select tools, e.g. a &quot;galaxy public data set&quot;. It also supports layering dataset products so that one data set might just be the metadata associated with hooking another data set into specific tools.<br /> <br /> This organization of apps and datasets helps us treat them as similar abstractions with similar management/curation/oversight demands. It also let's us map Galaxy's needs more clearly into an environment that is consistent across tools.</div>

Jpr@uab.edu