Anaconda: Difference between revisions
| Line 244: | Line 244: | ||
cd ~ | cd ~ | ||
mv ~/.conda $USER_DATA/ | mv ~/.conda $USER_DATA/ | ||
ln -s $USER_DATA/.conda | ln -s $USER_DATA/.conda .conda | ||
</pre> | </pre> | ||
Revision as of 18:21, 16 September 2020
Conda is a powerful package manager and environment manager. Conda allows you to maintain distinct environments for your different projects, with dependency packages defined and installed for each project.
Creating a Conda virtual environment
First step, direct conda to store files in $USER_DATA to avoid filling up $HOME. Create the $HOME/.condarc file by running the following code:
cat << "EOF" > ~/.condarc pkgs_dirs: - $USER_DATA/.conda/pkgs envs_dirs: - $USER_DATA/.conda/envs EOF
Load one of the conda environments available on Cheaha (Note, starting with Anaconda 2018.12, Anaconda releases changed to using YYYY.MM format for version numbers):
$ module -t avail Anaconda ... Anaconda3/5.3.0 Anaconda3/5.3.1 Anaconda3/2019.10
$ module load Anaconda3/2019.10
Once you have loaded Anaconda, you can create an environment using the following command (change test_env to whatever you want to name your environment):
$ conda create --name test_env
Solving environment: done
## Package Plan ##
environment location: /home/ravi89/.conda/envs/test_env
added / updated specs:
- setuptools
The following packages will be downloaded:
package | build
---------------------------|-----------------
python-3.7.0 | h6e4f718_3 30.6 MB
wheel-0.32.1 | py37_0 35 KB
setuptools-40.4.3 | py37_0 556 KB
------------------------------------------------------------
Total: 31.1 MB
The following NEW packages will be INSTALLED:
ca-certificates: 2018.03.07-0
certifi: 2018.8.24-py37_1
libedit: 3.1.20170329-h6b74fdf_2
libffi: 3.2.1-hd88cf55_4
libgcc-ng: 8.2.0-hdf63c60_1
libstdcxx-ng: 8.2.0-hdf63c60_1
ncurses: 6.1-hf484d3e_0
openssl: 1.0.2p-h14c3975_0
pip: 10.0.1-py37_0
python: 3.7.0-h6e4f718_3
readline: 7.0-h7b6447c_5
setuptools: 40.4.3-py37_0
sqlite: 3.25.2-h7b6447c_0
tk: 8.6.8-hbc83047_0
wheel: 0.32.1-py37_0
xz: 5.2.4-h14c3975_4
zlib: 1.2.11-ha838bed_2
Proceed ([y]/n)? y
Downloading and Extracting Packages
python-3.7.0 | 30.6 MB | ########################################################################### | 100%
wheel-0.32.1 | 35 KB | ########################################################################### | 100%
setuptools-40.4.3 | 556 KB | ########################################################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use:
# > source activate test_env
#
# To deactivate an active environment, use:
# > source deactivate
#
You can also specify the packages that you want to install in the conda virtual environment:
$ conda create --name test_env PACKAGE_NAME
Listing all your conda virtual environments
In case you forget the name of your virtual environments, you can list all your virtual environments by running conda env list
$ conda env list # conda environments: # jupyter_test /home/ravi89/.conda/envs/jupyter_test modeller /home/ravi89/.conda/envs/modeller psypy3 /home/ravi89/.conda/envs/psypy3 test /home/ravi89/.conda/envs/test test_env /home/ravi89/.conda/envs/test_env test_pytorch /home/ravi89/.conda/envs/test_pytorch tomopy /home/ravi89/.conda/envs/tomopy base * /share/apps/rc/software/Anaconda3/5.2.0 DeepNLP /share/apps/rc/software/Anaconda3/5.2.0/envs/DeepNLP ubrite-jupyter-base-1.0 /share/apps/rc/software/Anaconda3/5.2.0/envs/ubrite-jupyter-base-1.0
NOTE: Virtual environment with the asterisk(*) next to it is the one that's currently active.
Activating a conda virtual environment
You can activate your virtual environment for use by running source activate followed by conda activate ENV_NAME
$ source activate $ conda activate test_env (test_env) $
NOTE: Your shell prompt would also include the name of the virtual environment that you activated.
IMPORTANT!
source activate <env> is not idempotent. Using it twice with the same environment in a given session can lead to unexpected behavior. The recommended workflow is to use source activate to source the conda activate script, followed by conda activate <env>.
Locate and install packages
Conda allows you to search for packages that you want to install:
(test_env) $ conda search BeautifulSoup4 Loading channels: done # Name Version Build Channel beautifulsoup4 4.4.0 py27_0 pkgs/free beautifulsoup4 4.4.0 py34_0 pkgs/free beautifulsoup4 4.4.0 py35_0 pkgs/free ... beautifulsoup4 4.6.3 py35_0 pkgs/main beautifulsoup4 4.6.3 py36_0 pkgs/main beautifulsoup4 4.6.3 py37_0 pkgs/main (test_env) $
NOTE: Search is case-insensitive
You can install the packages in conda environment using
(test_env) $ conda install beautifulsoup4
Solving environment: done
## Package Plan ##
environment location: /home/ravi89/.conda/envs/test_env
added / updated specs:
- beautifulsoup4
The following packages will be downloaded:
package | build
---------------------------|-----------------
beautifulsoup4-4.6.3 | py37_0 138 KB
The following NEW packages will be INSTALLED:
beautifulsoup4: 4.6.3-py37_0
Proceed ([y]/n)? y
Downloading and Extracting Packages
beautifulsoup4-4.6.3 | 138 KB | ########################################################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
(test_env) $
Deactivating your virtual environment
You can deactivate your virtual environment using source deactivate
(test_env) $ source deactivate $
Sharing an environment
You may want to share your environment with someone for testing or other purposes. Sharing the environemnt file for your virtual environment is the most starightforward metohd which allows other person to quickly create an environment identical to you.
Export environment
- Activate the virtual environment that you want to export.
- Export an environment.yml file
conda env export -n test_env > environment.yml
- Now you can send the recently created environment.yml file to the other person.
Create a virtual environment using environment.yml
conda env create -f environment.yml -n test_env
Delete a conda virtual environment
You can use the remove parameter of conda to delete a conda virtual environment that you don't need:
$ conda remove --name test_env --all
Remove all packages in environment /home/ravi89/.conda/envs/test_env:
## Package Plan ##
environment location: /home/ravi89/.conda/envs/test_env
The following packages will be REMOVED:
beautifulsoup4: 4.6.3-py37_0
ca-certificates: 2018.03.07-0
certifi: 2018.8.24-py37_1
libedit: 3.1.20170329-h6b74fdf_2
libffi: 3.2.1-hd88cf55_4
libgcc-ng: 8.2.0-hdf63c60_1
libstdcxx-ng: 8.2.0-hdf63c60_1
ncurses: 6.1-hf484d3e_0
openssl: 1.0.2p-h14c3975_0
pip: 10.0.1-py37_0
python: 3.7.0-h6e4f718_3
readline: 7.0-h7b6447c_5
setuptools: 40.4.3-py37_0
sqlite: 3.25.2-h7b6447c_0
tk: 8.6.8-hbc83047_0
wheel: 0.32.1-py37_0
xz: 5.2.4-h14c3975_4
zlib: 1.2.11-ha838bed_2
Proceed ([y]/n)? y
Moving conda directory
As you build new conda environments, you may find that it is taking a lot of space in your $HOME directory. Here are 2 methods:
Method 1: Move a pre-existing conda directory and create a symlink
cd ~ mv ~/.conda $USER_DATA/ ln -s $USER_DATA/.conda .conda
Method 2: Create a "$HOME/.condarc" file in the $HOME directory by running the following code
cat << "EOF" > ~/.condarc pkgs_dirs: - $USER_DATA/.conda/pkgs envs_dirs: - $USER_DATA/.conda/envs EOF