FreeSurfer: Difference between revisions

From Cheaha
Jump to navigation Jump to search
(→‎Prep for running example recon_all process: add sections to prep for running recon-all)
 
(23 intermediate revisions by 5 users not shown)
Line 8: Line 8:
# Intersubject averaging of structural and functional data using a procedure that aligns individuals based on their cortical folding patterns for optimal alignment of homologous neural regions.
# Intersubject averaging of structural and functional data using a procedure that aligns individuals based on their cortical folding patterns for optimal alignment of homologous neural regions.


== Working with FreeSurfer on a Cluster ==
The [http://surfer.nmr.mgh.harvard.edu/fswiki/ReleaseNotes latest version of Freesurfer available is 5.3.0]. The last update to this software was 2 March 2013.
FreeSurfer run its process in a non-parallel environment, so there won't be much benefit from a dual/quad/hex core machine for a single case analysis. However a number of FreeSurfer recon-all processes can be started in the same machine, thereby, theoretically reducing the time to analyze a group of cases.


Here's how to run many recon-all processes on Cheaha.
== FreeSurfer on Cheaha ==
FreeSurfer-5.3 is installed on Cheaha. ''module'' command can be used to find out the current (latest) installed version. For FreeSurfer, use
<pre>
module whatis FreeSurfer
</pre>
 
FreeSurfer runs its process in a non-parallel environment, so there won't be much benefit from a dual/quad/hex core machine for a single case analysis. However, a number of FreeSurfer recon-all processes can be started in the same machine, thereby, theoretically reducing the time to analyze a group of cases. Following is a tutorial to setup your FreeSurfer environment and to run an example recon-all process on Cheaha.
 
=== SSH-key Config ===
 
Windows:
[http://docs.uabgrid.uab.edu/wiki/WorkingWithPutty#SSH_Keys Configure putty] to connect to Cheaha with ssh-keys.
 
Follow [http://docs.uabgrid.uab.edu/wiki/MatLab_SSH_Config#Linux this link] for ssh key config on Linux.
 
Follow [http://docs.uabgrid.uab.edu/wiki/MatLab_SSH_Config#Mac this link] for ssh key config on Mac.


=== Set up FreeSurfer Environment ===
=== Environment Setup ===


Setting up an environment to run FreeSurfer on Cheaha, consists of defining FreeSurfer related variables and their installation paths. Steps 1 and 2 are a one-time set up, ie., the first time you login to Cheaha. Need not set up and source env variables defined in .bashrc from the next time you login to Cheaha (as at login bashrc is auto-sourced).
For setting up an environment to run FreeSurfer on Cheaha, you need to load it's module to your environment:
1. To set up your environment on Cheaha, open $HOME/.bashrc with your favorite text editor such as vim, nano, emacs etc., and copy-paste the contents below:
<pre>
<pre>
export FREESURFER_HOME=/share/apps/freesurfer/4.5
module load rc/freesurfer/freesurfer-5.3.0
source $FREESURFER_HOME/SetUpFreeSurfer.sh
export FSFAST_HOME=$FREESURFER_HOME/fsfast
export FSF_OUTPUT_FORMAT=nii
export MNI_DIR=$FREESURFER_HOME/mni
export FSL_DIR=$HOME/fsl
export PATH=$FREESURFER_HOME/bin:$FSFAST_HOME/bin:$MNI_DIR/bin:$FSL_DIR/bin:$PATH
</pre>
</pre>
NOTE: In the above bashrc, FSL_DIR is also defined. FSL is not part of FreeSurfer, but needs to be installed separately. [http://www.fmrib.ox.ac.uk/fsldownloads/ Download FSL from here].


2. Source the above environment variables.
A lot of environment variables are set up, when you load FreeSurfer module, such as:
<pre>
<pre>
source $HOME/.bashrc
FREESURFER_HOME=/share/apps/freesurfer/5.3.0
FSFAST_HOME=$FREESURFER_HOME/fsfast
FUNCTIONALS_DIR=$FREESURFER_HOME/sessions
FSF_OUTPUT_FORMAT=nii
MNI_DIR=/share/apps/freesurfer/5.3.0/mni
MINC_BIN_DIR=$FREESURFER_HOME/mni/bin
MINC_LIB_DIR=$FREESURFER_HOME/mni/lib
MNI_DATAPATH=$FREESURFER_HOME/mni/data
FSL_DIR=$HOME/fsl
PERL5LIB $FREESURFER_HOME/mni/lib/perl5/5.8.5
FS_OVERRIDE 0
FMRI_ANALYSIS_DIR $FREESURFER_HOME/fsfast
PATH $FREESURFER_HOME/bin:$FREESURFER_HOME/fsfast/bin:$FREESURFER_HOME/mni/bin:/share/apps/freesurfer/QA_TOOLS
QA_TOOLS /share/apps/freesurfer/QA_TOOLS
</pre>
</pre>


=== Run example recon_all process ===
NOTE:
==== Get example image ====
* FSL_DIR is also defined when you load FreeSurfer module. FSL is not part of FreeSurfer, but is installed and available to all users on Cheaha. To start using FSL/bedpostx [http://docs.uabgrid.uab.edu/wiki/FSL see this documentation.]
First prepare by creating a subjects directory to hold image files in .nii format
* QA_TOOLS are also included, which are intended to be used in assessing the quality of one or more FreeSurfer recons. To use QA_TOOLS [https://surfer.nmr.mgh.harvard.edu/fswiki/QATools see this documentation.]
{{{
 
=== Example recon_all ===
 
1. Prepare by creating a subjects directory to hold image files in .nii format
 
<pre>
mkdir $HOME/subjects
mkdir $HOME/subjects
}}}
 
cd $HOME/subjects
</pre>
 
2. Get example image by copying the line below
 
<pre>
wget http://repo.rc.uab.edu/files/freesurfer/example.nii.gz
 
gunzip example.nii.gz
</pre>


==== Create job submission script ====
==== Create job submission script ====
This is a simple SGE script to submit recon-all process. This script first copies your subjects directory form your $HOME containing the MRI data to Cheaha's high performance file system, $UABGRID_SCRATCH, so as to ramp the speed of recon-all process. The recon-all process is then executed on $UABGRID_SCRATCH.


NOTE: Modify the SUBJECTS_DIR location to your local settings
This is a simple SGE script to submit recon-all process. This script first copies your subjects directory from your $HOME containing the MRI data to Cheaha's high performance file system, $USER_SCRATCH, so as to ramp the speed of recon-all process. The recon-all process is then executed on $USER_SCRATCH.


NOTE: For the example run, we've created a subjects directory at $HOME to hold the example.nii image.
For your actual run with real images, if you've copied your images to another location, then make sure to change $HOME/subjects (line no 7) accordingly in the script below.
Open a text editor and copy-paste the below contents:
<pre>
<pre>
#!/bin/bash
#!/bin/bash
scratch=$USER_SCRATCH
jobs=$scratch/jobs
sub=$scratch/subjects
mkdir $jobs
cp -r $HOME/subjects $scratch/


function run-recon-all() {
function run-recon-all() {
 
cd $scratch
for patient in `ls -1`
#create submit script for each patient in the subjects directory
for patient in `ls -1 $sub`
do
do
unset SUBJECTS_DIR
unset SUBJECTS_DIR
SUBJECTS_DIR=$UABGRID_SCRATCH/subjects
SUBJECTS_DIR=$sub
export SUBJECTS_DIR
export SUBJECTS_DIR
cat > submit-recon-all-$patient.sh <<EOF  
cat > $jobs/recon-all-$patient.sh <<EOF  
#!/bin/bash
#!/bin/bash
#$ -S/bin/bash
#$ -S /bin/bash
#$ -cwd
#$ -cwd
#
#
#$ -N recon-all-job$patient
#$ -N recon-all-$patient
# Set the hard and soft run time limits (ex: 1hour/58 min)
# Set the hard and soft run time limits (ex: 1hour/58 min)
#$ -l h_rt=30:00:00,s_rt=29:55:00
#$ -l h_rt=30:00:00,s_rt=29:55:00
#$ -j y
#$ -j y
#
#$ -M YOUR_EMAIL_ADDRESS
#$ -M YOUR_EMAIL_ADDRESS
#$ -m eas
#$ -m eas
# Load the appropriate module file(s) for your job
# set up FreeSurfer environment variables


mri_convert -oi -os $SUBJECTS_DIR/$patient.nii $SUBJECTS_DIR/$patient.mgz  
mri_convert -oi -os $SUBJECTS_DIR/$patient $SUBJECTS_DIR/$patient.mgz  


sleep 15
sleep 15
Line 76: Line 118:
recon-all -sd $SUBJECTS_DIR -s $patient
recon-all -sd $SUBJECTS_DIR -s $patient
EOF
EOF
done


qsub submit-recon-all-$patient.sh
#submit job script for each patient in the jobs directory to the cluster
 
pushd $jobs
for script in `ls -1`
do
echo "submitting job $script"
qsub $script
done
done
 
popd
}
}
cp -r $HOME/subjects $UABGRID_SCRATCH/
cd $UABGRID_SCRATCH/subjects
run-recon-all
run-recon-all
</pre>


</pre>
==== Make script executable ====


Make the above script executable
Make the above script executable
Line 95: Line 138:
chmod +x <name-of-script-above>
chmod +x <name-of-script-above>
</pre>
</pre>
==== Submit job script ====


Then submit that script with
Then submit that script with
Line 101: Line 146:
</pre>
</pre>


Check on it with qstat.
==== Track status of job ====
 
Check on the job status with squeue.
<pre>
<pre>
qstat -u $USER
squeue -u $USER
</pre>
</pre>


After the completion of the recon-all process, copy the results back to your $HOME and delete the contents from scratch
==== Clean-up scratch ====
 
After the completion of the recon-all process, copy the results and logs back to your $HOME and delete the contents from scratch
<pre>
<pre>
cp -r $UABGRID_SCRATCH/subjects $HOME/subjects-results
cp -r $USER_SCRATCH/subjects $HOME/subjects-results
 
cp -r $USER_SCRATCH/jobs $HOME/jobs-results


rm -rf $UABGRID_SCRATCH/subjects
rm -rf $USER_SCRATCH/subjects
 
rm -rf $USER_SCRATCH/jobs
</pre>
</pre>
== Transfer files from Cheaha to local computer ==
Transfer the subjects/jobs directory from your home account on Cheaha to your local computer with '''psftp'''. [http://docs.uabgrid.uab.edu/wiki/WorkingWithPutty#File_Transfer Here's how to transfer files using psftp].
== References ==
[http://surfer.nmr.mgh.harvard.edu/fswiki/TestingFreeSurfer Freesurfer wiki]
[http://surfer.nmr.mgh.harvard.edu/fswiki/FsTutorial/Data Freesurfer Tutorial]
[http://surfer.nmr.mgh.harvard.edu/fswiki/recon-all recon-all]
[http://surfer.nmr.mgh.harvard.edu/fswiki/mri_convert mri_convert]
[[Category:Software]][[Category:Image Processing]]

Latest revision as of 17:29, 18 November 2016

Overview

FreeSurfer is a freely available software package developed by investigators at the Athinoula A. Martinos Center for Biomedical Imaging used for a number of procedures including:

  1. Creation of computerized models of the brain from magnetic resonance imaging (MRI) data.
  2. Processing of functional magnetic resonance imaging (fMRI) data.
  3. Measuring a number of morphometric properties of the brain including cortical thickness and regional volumes.
  4. Intersubject averaging of structural and functional data using a procedure that aligns individuals based on their cortical folding patterns for optimal alignment of homologous neural regions.

The latest version of Freesurfer available is 5.3.0. The last update to this software was 2 March 2013.

FreeSurfer on Cheaha

FreeSurfer-5.3 is installed on Cheaha. module command can be used to find out the current (latest) installed version. For FreeSurfer, use

module whatis FreeSurfer

FreeSurfer runs its process in a non-parallel environment, so there won't be much benefit from a dual/quad/hex core machine for a single case analysis. However, a number of FreeSurfer recon-all processes can be started in the same machine, thereby, theoretically reducing the time to analyze a group of cases. Following is a tutorial to setup your FreeSurfer environment and to run an example recon-all process on Cheaha.

SSH-key Config

Windows: Configure putty to connect to Cheaha with ssh-keys.

Follow this link for ssh key config on Linux.

Follow this link for ssh key config on Mac.

Environment Setup

For setting up an environment to run FreeSurfer on Cheaha, you need to load it's module to your environment:

module load rc/freesurfer/freesurfer-5.3.0

A lot of environment variables are set up, when you load FreeSurfer module, such as:

FREESURFER_HOME=/share/apps/freesurfer/5.3.0 
FSFAST_HOME=$FREESURFER_HOME/fsfast 
FUNCTIONALS_DIR=$FREESURFER_HOME/sessions 
FSF_OUTPUT_FORMAT=nii 
MNI_DIR=/share/apps/freesurfer/5.3.0/mni 
MINC_BIN_DIR=$FREESURFER_HOME/mni/bin 
MINC_LIB_DIR=$FREESURFER_HOME/mni/lib 
MNI_DATAPATH=$FREESURFER_HOME/mni/data 
FSL_DIR=$HOME/fsl 
PERL5LIB $FREESURFER_HOME/mni/lib/perl5/5.8.5 
FS_OVERRIDE 0 
FMRI_ANALYSIS_DIR $FREESURFER_HOME/fsfast 
PATH $FREESURFER_HOME/bin:$FREESURFER_HOME/fsfast/bin:$FREESURFER_HOME/mni/bin:/share/apps/freesurfer/QA_TOOLS 
QA_TOOLS /share/apps/freesurfer/QA_TOOLS 

NOTE:

  • FSL_DIR is also defined when you load FreeSurfer module. FSL is not part of FreeSurfer, but is installed and available to all users on Cheaha. To start using FSL/bedpostx see this documentation.
  • QA_TOOLS are also included, which are intended to be used in assessing the quality of one or more FreeSurfer recons. To use QA_TOOLS see this documentation.

Example recon_all

1. Prepare by creating a subjects directory to hold image files in .nii format

mkdir $HOME/subjects

cd $HOME/subjects

2. Get example image by copying the line below

wget http://repo.rc.uab.edu/files/freesurfer/example.nii.gz

gunzip example.nii.gz

Create job submission script

This is a simple SGE script to submit recon-all process. This script first copies your subjects directory from your $HOME containing the MRI data to Cheaha's high performance file system, $USER_SCRATCH, so as to ramp the speed of recon-all process. The recon-all process is then executed on $USER_SCRATCH.

NOTE: For the example run, we've created a subjects directory at $HOME to hold the example.nii image. For your actual run with real images, if you've copied your images to another location, then make sure to change $HOME/subjects (line no 7) accordingly in the script below.

Open a text editor and copy-paste the below contents:

#!/bin/bash

scratch=$USER_SCRATCH
jobs=$scratch/jobs
sub=$scratch/subjects
mkdir $jobs
cp -r $HOME/subjects $scratch/

function run-recon-all() {
cd $scratch
#create submit script for each patient in the subjects directory
for patient in `ls -1 $sub`
do
unset SUBJECTS_DIR
SUBJECTS_DIR=$sub
export SUBJECTS_DIR
cat > $jobs/recon-all-$patient.sh <<EOF 
#!/bin/bash
#$ -S /bin/bash
#$ -cwd
#
#$ -N recon-all-$patient
# Set the hard and soft run time limits (ex: 1hour/58 min)
#$ -l h_rt=30:00:00,s_rt=29:55:00
#$ -j y
#$ -M YOUR_EMAIL_ADDRESS
#$ -m eas
# set up FreeSurfer environment variables

mri_convert -oi -os $SUBJECTS_DIR/$patient $SUBJECTS_DIR/$patient.mgz 

sleep 15

recon-all -sd $SUBJECTS_DIR -s $patient
EOF
done

#submit job script for each patient in the jobs directory to the cluster
pushd $jobs
for script in `ls -1`
do
echo "submitting job $script"
qsub $script
done
popd
}
run-recon-all

Make script executable

Make the above script executable

chmod +x <name-of-script-above>

Submit job script

Then submit that script with

./name-of-script-above

Track status of job

Check on the job status with squeue.

squeue -u $USER

Clean-up scratch

After the completion of the recon-all process, copy the results and logs back to your $HOME and delete the contents from scratch

cp -r $USER_SCRATCH/subjects $HOME/subjects-results

cp -r $USER_SCRATCH/jobs $HOME/jobs-results

rm -rf $USER_SCRATCH/subjects

rm -rf $USER_SCRATCH/jobs

Transfer files from Cheaha to local computer

Transfer the subjects/jobs directory from your home account on Cheaha to your local computer with psftp. Here's how to transfer files using psftp.

References

Freesurfer wiki

Freesurfer Tutorial

recon-all

mri_convert