Open OnDemand Sandbox: Difference between revisions

From Cheaha
Jump to navigation Jump to search
No edit summary
 
(7 intermediate revisions by 3 users not shown)
Line 3: Line 3:
===Setting up sandbox for your profile===
===Setting up sandbox for your profile===
Setting up sandbox is pretty simple and straightforward.  
Setting up sandbox is pretty simple and straightforward.  
* You need to create a dev folder in your '''DATA_USER/ondemand''' folder
* You need to create a dev folder in your '''$USER_DATA/ondemand''' folder
<pre>
<pre>
mkdir -p /data/user/$USER/ondemand/dev
mkdir -p $USER_DATA/ondemand/dev
</pre>
</pre>


Line 12: Line 12:
* You have successfully enabled sandbox for your environment, and you should see a new '''Develop''' tab.
* You have successfully enabled sandbox for your environment, and you should see a new '''Develop''' tab.


'''NOTE:''' This is a one time setup


===Building your first dev app===
===Building your first dev app===


Over here we provide instructions to create your first example dev script:
Over here you have instructions to create your first example dev app:
 
====IGV====


* Under '''Develop''' tab , click on '''My SandBox Apps (Development)'''
* Under '''Develop''' tab , click on '''My SandBox Apps (Development)'''
Line 26: Line 30:
<pre>
<pre>
Directory Name: IGV
Directory Name: IGV
Git Remote: https://github.com/uabrc/ood_igv.git
Git Remote: https://github.com/uabrc/sandbox_app_template
</pre>
</pre>
and '''Submit'''
and '''Submit'''
Line 32: Line 36:
'''Note:''' You can also check the line, about creating a new git repo, if you want to use it to maintain your own instance of the app.
'''Note:''' You can also check the line, about creating a new git repo, if you want to use it to maintain your own instance of the app.


* Now click on '''Files''' to take a look at the files for the app (git repo), that you just cloned.
* Now click on '''Files''' to take a look at the files for the app (local git repo working directory), that you just cloned.


* Highlight '''form.yml''' and click on '''Edit''' . Change the file to the following content:
* Let's modify '''manifest.yml''' file (highlight the file, and click on '''edit''') to change metadata for the sandbox app. Change the SOFTWARE_NAME and SOFTWARE_LINK variables appropriately.
In this case change then to following:
<pre>
<pre>
---
SOFTWARE_NAME : IGV
cluster: "SLURM_CLUSTER"
SOFTWARE_LINK : http://software.broadinstitute.org/software/igv/
attributes:
</pre>
  bc_num_hours:
    value: 1


  bc_num_slots:
* Next, highlight '''form.yml''' and click on '''Edit''' . Uncomment version section in the file, and change that section as follows:
    label: Number of CPU
    value: 1
    min: 1
    max: 24
    step: 1


  bc_num_mems:
Change '''SOFTWARE_NAME''' appropriately. Under options change '''display_version''' to the the version that you want to display on the form, and change '''corresponding_cheaha_module''' to matching module name on cheaha.
    widget: "number_field"
    label: Memory per CPU (GB)
    value: 4
    min: 1
    max: 128
    step: 1
 
  bc_partition:
    widget: select
    label: Partition
    options:
      - [ "express", "express" ]
      - [ "short", "short" ]
      - [ "medium", "medium" ]
      - [ "long", "long" ]
      - [ "interactive", "interactive" ]
      - [ "pascalnodes", "pascalnodes" ]


Here we are specifying what the form for the app is going to look like, and changing available parameters for Cheaha.
<pre>
   version:
   version:
     widget: select
     widget: select
Line 74: Line 57:
     options:
     options:
       - [ "2.5.0-Java-11", "IGV/2.5.0-Java-11" ]
       - [ "2.5.0-Java-11", "IGV/2.5.0-Java-11" ]
form:
  - version
  - bc_num_hours
  - bc_partition
  - bc_num_slots
  - bc_num_mems
  - bc_email_on_started
</pre>
</pre>
and save the file, and close the tab.
and save the file, and close the tab.


Here we are specifying what the form for the app is going to look like, and changing available parameters for Cheaha.
The "native" section in '''submit.yml.erb''' includes the slurm parameters you would use with sbatch and describes the resources needed for the app.  These can include variable names defined in the form above which will get substituted for the user provided values when the app is run.
 
* Now again in the file explorer for your app, click on '''New File''' with name '''submit.yml.erb'''


* Edit the file submit.yml.erb, with the following content.
* Now, go into '''template''' and edit file '''script.sh.erb''', by adding following lines at the end of the file.  This is the actual job script that will be submitted to the cluster, just like you would submit with sbatch from the command line.
<pre>
<pre>
---
igvtools_gui.command &
batch_connect:
igv.sh
  template: "vnc"
script:
  native:
    - "-N 1"
    - "-n <%= bc_num_slots.blank? ? 1 : bc_num_slots.to_i %>"
    - "--mem-per-cpu=<%= bc_num_mems.blank? ? 4 : bc_num_mems.to_i %>G"
    - "--partition=<%= bc_partition %>"
    - "--time=<%= bc_num_hours.blank? ? 1 : bc_num_hours.to_i %>:00:00"
    - "--job-name=ood-igv"
<%- if bc_partition == "pascalnodes" -%>
    - "--gres=gpu:1"
<%- end -%>
</pre>
</pre>
SAve the file, and close the tab
Save the file, and close the tab.
This script runs the app executable after loading the appropriate module, based on the version you choose in the form.


* Now, go into '''template''' folder and create a new file '''script.sh.erb''', and add following content to it:
* Close the file browser for your app.
<pre>
#!/usr/bin/env bash


* Click on '''Launch IGV''', and submit the job with desired parameters.


# Clean the environment
You should have your first dev app (IGV) running.
module reset


# Set working directory to home directory
cd "${HOME}"


#
====FSL====
# Launch Xfce Window Manager and Panel
* Under '''Develop''' tab , click on '''My SandBox Apps (Development)'''
#


(
* Now click on '''New App''' tab
  export SEND_256_COLORS_TO_REMOTE=1
  export XDG_CONFIG_HOME="<%= session.staged_root.join("config") %>"
  export XDG_DATA_HOME="<%= session.staged_root.join("share") %>"
  export XDG_CACHE_HOME="$(mktemp -d)"
  module restore
  set -x
  xfwm4 --compositor=off --daemon --sm-client-disable
  xsetroot -solid "#D3D3D3"
  xfsettingsd --sm-client-disable
  xfce4-panel --sm-client-disable
) &


#
* Now we are going to import an already existing git repo for an app, by clicking on '''Clone Existing App'''
# Start Integrative Genomics Viewer
#


# Load the required environment
* Add the following values in the form:
module load <%= context.version %>
<pre>
# Launch IGV
Directory Name: FSL
module list
Git Remote: https://github.com/uabrc/sandbox_app_template
set -x
</pre>
igvtools_gui.command &
and '''Submit'''
igv.sh
 
'''Note:''' You can also check the line, about creating a new git repo, if you want to use it to maintain your own instance of the app.
 
* Now click on '''Files''' to take a look at the files for the app (local git repo working directory), that you just cloned.
 
* Let's modify '''manifest.yml''' file (highlight the file, and click on '''edit''') to change metadata for the sandbox app. Change the SOFTWARE_NAME and SOFTWARE_LINK variables appropriately.
In this case change then to following:
<pre>
SOFTWARE_NAME : FSL
SOFTWARE_LINK : https://www.fmrib.ox.ac.uk/fsl
</pre>
</pre>
Save the file, and close the tab. Now click on '''Open in terminal''', and run
 
* Next, highlight '''form.yml''' and click on '''Edit''' . Uncomment version section in the file, and change that section as follows:
<pre>
<pre>
chmod +x script.sh.erb
  version:
    widget: select
    label: "FSL version"
    help: "This defines the version of FSL you want to load."
    options:
      - [ "6.0.3", "FSL/6.0.3" ]
</pre>
</pre>
This script needs to be an executable file.
and save the file, and close the tab.


This script runs the executable after loading the appropriate module, absed on the version you choose in the form.
Change '''SOFTWARE_NAME''' appropriately. Under options change '''display_version''' to the the version that you want to display on the form, and change '''corresponding_cheaha_module''' to matching module name on cheaha.
 
Here we are specifying what the form for the app is going to look like, and changing available parameters for Cheaha.
 
The "native" section in '''submit.yml.erb''' includes the slurm parameters you would use with sbatch and describes the resources needed for the app.  These can include variable names defined in the form above which will get substituted for the user provided values when the app is run.
 
* Now, go into '''template''' and edit file '''script.sh.erb''', by adding following lines at the end of the file.  This is the actual job script that will be submitted to the cluster, just like you would submit with sbatch from the command line.
<pre>
source /share/apps/rc/software/FSL/6.0.3/fsl/etc/fslconf/fsl.sh
fsl
</pre>
Save the file, and close the tab.
This script runs the app executable after loading the appropriate module, based on the version you choose in the form.


* Now close the file browser for your app.
* Close the file browser for your app.


* Now click, on 'Launch IGV', and submit the job.
* Click on '''Launch IGV''', and submit the job with desired parameters.


You should have your first dev app running.  
You should have your first dev app (IGV) running.


===More info===
===More info===

Latest revision as of 22:44, 11 August 2020

Open OnDemand is a web interface to Cheaha HPC resources. We have setup multiple general purpose apps, such as HPC Desktop, Jupyter, RStudio etc, on cheaha, to make the process of accessing these basic softwares easier. OnDemand also allows users to setup their own personal apps in the OnDemand, as sandbox apps.

Setting up sandbox for your profile

Setting up sandbox is pretty simple and straightforward.

  • You need to create a dev folder in your $USER_DATA/ondemand folder
mkdir -p $USER_DATA/ondemand/dev
  • Then go to Cheaha OnDemand , login, and under Help, click on Restart Web Server
  • You have successfully enabled sandbox for your environment, and you should see a new Develop tab.


NOTE: This is a one time setup

Building your first dev app

Over here you have instructions to create your first example dev app:

IGV

  • Under Develop tab , click on My SandBox Apps (Development)
  • Now click on New App tab
  • Now we are going to import an already existing git repo for an app, by clicking on Clone Existing App
  • Add the following values in the form:
Directory Name: IGV
Git Remote: https://github.com/uabrc/sandbox_app_template

and Submit

Note: You can also check the line, about creating a new git repo, if you want to use it to maintain your own instance of the app.

  • Now click on Files to take a look at the files for the app (local git repo working directory), that you just cloned.
  • Let's modify manifest.yml file (highlight the file, and click on edit) to change metadata for the sandbox app. Change the SOFTWARE_NAME and SOFTWARE_LINK variables appropriately.

In this case change then to following:

SOFTWARE_NAME : IGV
SOFTWARE_LINK : http://software.broadinstitute.org/software/igv/
  • Next, highlight form.yml and click on Edit . Uncomment version section in the file, and change that section as follows:

Change SOFTWARE_NAME appropriately. Under options change display_version to the the version that you want to display on the form, and change corresponding_cheaha_module to matching module name on cheaha.

Here we are specifying what the form for the app is going to look like, and changing available parameters for Cheaha.

  version:
    widget: select
    label: "IGV version"
    help: "This defines the version of IGV you want to load."
    options:
      - [ "2.5.0-Java-11", "IGV/2.5.0-Java-11" ]

and save the file, and close the tab.

The "native" section in submit.yml.erb includes the slurm parameters you would use with sbatch and describes the resources needed for the app. These can include variable names defined in the form above which will get substituted for the user provided values when the app is run.

  • Now, go into template and edit file script.sh.erb, by adding following lines at the end of the file. This is the actual job script that will be submitted to the cluster, just like you would submit with sbatch from the command line.
igvtools_gui.command &
igv.sh

Save the file, and close the tab. This script runs the app executable after loading the appropriate module, based on the version you choose in the form.

  • Close the file browser for your app.
  • Click on Launch IGV, and submit the job with desired parameters.

You should have your first dev app (IGV) running.


FSL

  • Under Develop tab , click on My SandBox Apps (Development)
  • Now click on New App tab
  • Now we are going to import an already existing git repo for an app, by clicking on Clone Existing App
  • Add the following values in the form:
Directory Name: FSL
Git Remote: https://github.com/uabrc/sandbox_app_template

and Submit

Note: You can also check the line, about creating a new git repo, if you want to use it to maintain your own instance of the app.

  • Now click on Files to take a look at the files for the app (local git repo working directory), that you just cloned.
  • Let's modify manifest.yml file (highlight the file, and click on edit) to change metadata for the sandbox app. Change the SOFTWARE_NAME and SOFTWARE_LINK variables appropriately.

In this case change then to following:

SOFTWARE_NAME : FSL
SOFTWARE_LINK : https://www.fmrib.ox.ac.uk/fsl
  • Next, highlight form.yml and click on Edit . Uncomment version section in the file, and change that section as follows:
  version:
    widget: select
    label: "FSL version"
    help: "This defines the version of FSL you want to load."
    options:
      - [ "6.0.3", "FSL/6.0.3" ]

and save the file, and close the tab.

Change SOFTWARE_NAME appropriately. Under options change display_version to the the version that you want to display on the form, and change corresponding_cheaha_module to matching module name on cheaha.

Here we are specifying what the form for the app is going to look like, and changing available parameters for Cheaha.

The "native" section in submit.yml.erb includes the slurm parameters you would use with sbatch and describes the resources needed for the app. These can include variable names defined in the form above which will get substituted for the user provided values when the app is run.

  • Now, go into template and edit file script.sh.erb, by adding following lines at the end of the file. This is the actual job script that will be submitted to the cluster, just like you would submit with sbatch from the command line.
source /share/apps/rc/software/FSL/6.0.3/fsl/etc/fslconf/fsl.sh
fsl

Save the file, and close the tab. This script runs the app executable after loading the appropriate module, based on the version you choose in the form.

  • Close the file browser for your app.
  • Click on Launch IGV, and submit the job with desired parameters.

You should have your first dev app (IGV) running.

More info

To get more example apps to build or to build passenger apps, look here.


Contact us if you run into any issues, or to discuss any new app.