Anaconda: Difference between revisions

From Cheaha
Jump to navigation Jump to search
 
(9 intermediate revisions by 5 users not shown)
Line 2: Line 2:


===Creating a Conda virtual environment===
===Creating a Conda virtual environment===
Load one of the conda environments available on cheaha:
First step, direct conda to store files in $USER_DATA to avoid filling up $HOME. Create the '''$HOME/.condarc''' file by running the following code:
<pre>
<pre>
ravi89 @ c0066 ➜  ~ module avail Anaconda
cat << "EOF" > ~/.condarc
pkgs_dirs:
  - $USER_DATA/.conda/pkgs
envs_dirs:
  - $USER_DATA/.conda/envs
EOF
</pre>
 
Load one of the conda environments available on Cheaha (Note, starting with Anaconda 2018.12, Anaconda releases changed to using YYYY.MM format for version numbers):
<pre>
$ module -t avail Anaconda
...
Anaconda3/5.3.0
Anaconda3/5.3.1
Anaconda3/2019.10


--------------------------------------------- /share/apps/rc/modules/all ---------------------------------------------
Anaconda2/4.0.0 Anaconda2/4.2.0 Anaconda3/4.4.0 Anaconda3/5.0.1 Anaconda3/5.1.0 Anaconda3/5.2.0
</pre>
</pre>
Once you have loaded Anaconda, you can create an environment using the following command:
<pre>
<pre>
ravi89 @ c0066 ➜  ~ conda create --name test_env
$ module load Anaconda3/2019.10
</pre>
 
Once you have loaded Anaconda, you can create an environment using the following command (change '''test_env''' to whatever you want to name your environment):
<pre>
$ conda create --name test_env
 
Solving environment: done
Solving environment: done


## Package Plan ##
## Package Plan ##


   environment location: /home/ravi89/.conda/envs/test_env
   environment location: ~/.conda/envs/test_env


   added / updated specs:
   added / updated specs:
Line 53: Line 70:


Proceed ([y]/n)? y
Proceed ([y]/n)? y


Downloading and Extracting Packages
Downloading and Extracting Packages
Line 73: Line 89:
You can also specify the packages that you want to install in the conda virtual environment:
You can also specify the packages that you want to install in the conda virtual environment:
<pre>
<pre>
ravi89 @ c0066 ➜  ~ conda create --name test_env PACKAGE_NAME
$ conda create --name test_env PACKAGE_NAME
</pre>
</pre>


Line 79: Line 95:
In case you forget the name of your virtual environments, you can list all your virtual environments by running '''conda env list'''
In case you forget the name of your virtual environments, you can list all your virtual environments by running '''conda env list'''
<pre>
<pre>
ravi89 @ c0066 ➜  ~ conda env list
$ conda env list
# conda environments:
# conda environments:
#
#
jupyter_test            /home/ravi89/.conda/envs/jupyter_test
jupyter_test            ~/.conda/envs/jupyter_test
modeller                /home/ravi89/.conda/envs/modeller
modeller                ~/.conda/envs/modeller
psypy3                  /home/ravi89/.conda/envs/psypy3
psypy3                  ~/.conda/envs/psypy3
test                    /home/ravi89/.conda/envs/test
test                    ~/.conda/envs/test
test_env                /home/ravi89/.conda/envs/test_env
test_env                ~/.conda/envs/test_env
test_pytorch            /home/ravi89/.conda/envs/test_pytorch
test_pytorch            ~/.conda/envs/test_pytorch
tomopy                  /home/ravi89/.conda/envs/tomopy
tomopy                  ~/.conda/envs/tomopy
base                  *  /share/apps/rc/software/Anaconda3/5.2.0
base                  *  /share/apps/rc/software/Anaconda3/5.2.0
DeepNLP                  /share/apps/rc/software/Anaconda3/5.2.0/envs/DeepNLP
DeepNLP                  /share/apps/rc/software/Anaconda3/5.2.0/envs/DeepNLP
ubrite-jupyter-base-1.0    /share/apps/rc/software/Anaconda3/5.2.0/envs/ubrite-jupyter-base-1.0
ubrite-jupyter-base-1.0    /share/apps/rc/software/Anaconda3/5.2.0/envs/ubrite-jupyter-base-1.0
ravi89 @ c0066 ➜  ~
</pre>
</pre>
NOTE: Virtual environment with the asterisk(*) next to it is the one that's currently active.
NOTE: Virtual environment with the asterisk(*) next to it is the one that's currently active.


===Activating a conda virtual environment===
===Activating a conda virtual environment===
You can activate your virtual environment for use by running '''source activate ENV_NAME'''
You can activate your virtual environment for use by running '''source activate''' followed by '''conda activate ENV_NAME'''
 
<pre>
<pre>
ravi89 @ c0066 ➜  ~ source activate test_env
$ source activate
(test_env) ravi89 @ c0066 ➜  ~
$ conda activate test_env
(test_env) $
</pre>
</pre>
NOTE: Your shell prompt would also include the name of the virtual environment that you activated.
NOTE: Your shell prompt would also include the name of the virtual environment that you activated.
'''IMPORTANT!'''
The following only applies to versions prior to 2019.10. '''source activate <env>''' is not idempotent. Using it twice with the same environment in a given session can lead to unexpected behavior. The recommended workflow is to use '''source activate''' to source the '''conda activate''' script, followed by '''conda activate <env>'''.
From version 2019.10 and on, simply use '''conda activate <env>'''.


===Locate and install packages===
===Locate and install packages===
Conda allows you to search for packages that you want to install:
Conda allows you to search for packages that you want to install:
<pre>
<pre>
(test_env) ravi89 @ c0066 ➜  ~ conda search BeautifulSoup4
(test_env) $ conda search BeautifulSoup4
Loading channels: done
Loading channels: done
# Name                  Version          Build  Channel
# Name                  Version          Build  Channel
Line 114: Line 138:
beautifulsoup4            4.4.0          py34_0  pkgs/free
beautifulsoup4            4.4.0          py34_0  pkgs/free
beautifulsoup4            4.4.0          py35_0  pkgs/free
beautifulsoup4            4.4.0          py35_0  pkgs/free
beautifulsoup4            4.4.1          py27_0  pkgs/free
...
beautifulsoup4            4.4.1          py34_0  pkgs/free
beautifulsoup4            4.4.1          py35_0  pkgs/free
beautifulsoup4            4.5.1          py27_0  pkgs/free
beautifulsoup4            4.5.1          py34_0  pkgs/free
beautifulsoup4            4.5.1          py35_0  pkgs/free
beautifulsoup4            4.5.1          py36_0  pkgs/free
beautifulsoup4            4.5.3          py27_0  pkgs/free
beautifulsoup4            4.5.3          py34_0  pkgs/free
beautifulsoup4            4.5.3          py35_0  pkgs/free
beautifulsoup4            4.5.3          py36_0  pkgs/free
beautifulsoup4            4.6.0          py27_0  pkgs/free
beautifulsoup4            4.6.0          py27_1  pkgs/main
beautifulsoup4            4.6.0  py27h3f86ba9_1  pkgs/main
beautifulsoup4            4.6.0          py34_0  pkgs/free
beautifulsoup4            4.6.0          py35_0  pkgs/free
beautifulsoup4            4.6.0  py35h442a8c9_1  pkgs/main
beautifulsoup4            4.6.0          py36_0  pkgs/free
beautifulsoup4            4.6.0          py36_1  pkgs/main
beautifulsoup4            4.6.0  py36h49b8c8c_1  pkgs/main
beautifulsoup4            4.6.0          py37_1  pkgs/main
beautifulsoup4            4.6.1          py27_0  pkgs/main
beautifulsoup4            4.6.1          py35_0  pkgs/main
beautifulsoup4            4.6.1          py36_0  pkgs/main
beautifulsoup4            4.6.1          py37_0  pkgs/main
beautifulsoup4            4.6.3          py27_0  pkgs/main
beautifulsoup4            4.6.3          py35_0  pkgs/main
beautifulsoup4            4.6.3          py35_0  pkgs/main
beautifulsoup4            4.6.3          py36_0  pkgs/main
beautifulsoup4            4.6.3          py36_0  pkgs/main
beautifulsoup4            4.6.3          py37_0  pkgs/main
beautifulsoup4            4.6.3          py37_0  pkgs/main
(test_env) ravi89 @ c0066 ➜  ~
(test_env) $
</pre>
</pre>
NOTE: Search is case-insensitive
NOTE: Search is case-insensitive
Line 149: Line 148:
You can install the packages in conda environment using
You can install the packages in conda environment using
<pre>
<pre>
(test_env) ravi89 @ c0066 ➜  ~ conda install beautifulsoup4
(test_env) $ conda install beautifulsoup4
Solving environment: done
Solving environment: done


## Package Plan ##
## Package Plan ##


   environment location: /home/ravi89/.conda/envs/test_env
   environment location: ~/.conda/envs/test_env


   added / updated specs:
   added / updated specs:
Line 178: Line 177:
Verifying transaction: done
Verifying transaction: done
Executing transaction: done
Executing transaction: done
(test_env) ravi89 @ c0066 ➜  ~
(test_env) $
</pre>
</pre>


Line 184: Line 183:
You can deactivate your virtual environment using '''source deactivate'''
You can deactivate your virtual environment using '''source deactivate'''
<pre>
<pre>
(test_env) ravi89 @ c0066 ➜  ~ source deactivate
(test_env) $ source deactivate
ravi89 @ c0066 ➜  ~
$
</pre>
</pre>


Line 200: Line 199:
====Create a virtual environment using environment.yml====
====Create a virtual environment using environment.yml====
<pre>
<pre>
conda env create -f environment.yml
conda env create -f environment.yml -n test_env
</pre>
</pre>


===Delete a conda virtual environment===
===Delete a conda virtual environment===
You can use remove parameter of conda to delete a conda virtual environment that you don't need:
You can use the '''remove''' parameter of conda to delete a conda virtual environment that you don't need:
<pre>
<pre>
ravi89 @ c0066 ➜  ~ conda remove --name test_env --all
$ conda remove --name test_env --all


Remove all packages in environment /home/ravi89/.conda/envs/test_env:
Remove all packages in environment ~/.conda/envs/test_env:




## Package Plan ##
## Package Plan ##


   environment location: /home/ravi89/.conda/envs/test_env
   environment location: ~/.conda/envs/test_env




Line 238: Line 237:


Proceed ([y]/n)? y
Proceed ([y]/n)? y
ravi89 @ c0066 ➜  ~
</pre>
</pre>


===Moving conda directory===
===Moving conda directory===
As you build new conda environments, you may find that it is taking a lot of space in your $HOME directory. Here are 2 methods:
As you build new conda environments, you may find that it is taking a lot of space in your $HOME directory, leading to issues with interactive sessions failing to start. Below are 2 methods to resolve the issue.


Method 1: Move a pre-existing conda directory and create a symlink
Method 1: Move a pre-existing conda directory and create a symlink
<pre>
<pre>
cd
cd ~
mv ~/.conda $USER_DATA/
mv ~/.conda $USER_DATA/
ln -s $USER_DATA/.conda
ln -s $USER_DATA/.conda .conda
</pre>
</pre>


Method 2: Create a ".condarc" file in the $HOME directory containing the following
Method 2: Create a "$HOME/.condarc" file in the $HOME directory by running the following code
<pre>
<pre>
cat << "EOF" > ~/.condarc
pkgs_dirs:
pkgs_dirs:
   - $USER_DATA/.conda/pkgs
   - $USER_DATA/.conda/pkgs
envs_dirs:
envs_dirs:
   - $USER_DATA/.conda/envs
   - $USER_DATA/.conda/envs
EOF
</pre>
</pre>

Latest revision as of 19:50, 24 March 2021

Conda is a powerful package manager and environment manager. Conda allows you to maintain distinct environments for your different projects, with dependency packages defined and installed for each project.

Creating a Conda virtual environment

First step, direct conda to store files in $USER_DATA to avoid filling up $HOME. Create the $HOME/.condarc file by running the following code:

cat << "EOF" > ~/.condarc
pkgs_dirs:
  - $USER_DATA/.conda/pkgs
envs_dirs:
  - $USER_DATA/.conda/envs
EOF

Load one of the conda environments available on Cheaha (Note, starting with Anaconda 2018.12, Anaconda releases changed to using YYYY.MM format for version numbers):

$ module -t avail Anaconda
...
Anaconda3/5.3.0
Anaconda3/5.3.1
Anaconda3/2019.10

$ module load Anaconda3/2019.10 

Once you have loaded Anaconda, you can create an environment using the following command (change test_env to whatever you want to name your environment):

$ conda create --name test_env

Solving environment: done

## Package Plan ##

  environment location: ~/.conda/envs/test_env

  added / updated specs:
    - setuptools


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    python-3.7.0               |       h6e4f718_3        30.6 MB
    wheel-0.32.1               |           py37_0          35 KB
    setuptools-40.4.3          |           py37_0         556 KB
    ------------------------------------------------------------
                                           Total:        31.1 MB

The following NEW packages will be INSTALLED:

    ca-certificates: 2018.03.07-0
    certifi:         2018.8.24-py37_1
    libedit:         3.1.20170329-h6b74fdf_2
    libffi:          3.2.1-hd88cf55_4
    libgcc-ng:       8.2.0-hdf63c60_1
    libstdcxx-ng:    8.2.0-hdf63c60_1
    ncurses:         6.1-hf484d3e_0
    openssl:         1.0.2p-h14c3975_0
    pip:             10.0.1-py37_0
    python:          3.7.0-h6e4f718_3
    readline:        7.0-h7b6447c_5
    setuptools:      40.4.3-py37_0
    sqlite:          3.25.2-h7b6447c_0
    tk:              8.6.8-hbc83047_0
    wheel:           0.32.1-py37_0
    xz:              5.2.4-h14c3975_4
    zlib:            1.2.11-ha838bed_2

Proceed ([y]/n)? y

Downloading and Extracting Packages
python-3.7.0         | 30.6 MB   | ########################################################################### | 100%
wheel-0.32.1         | 35 KB     | ########################################################################### | 100%
setuptools-40.4.3    | 556 KB    | ########################################################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use:
# > source activate test_env
#
# To deactivate an active environment, use:
# > source deactivate
#

You can also specify the packages that you want to install in the conda virtual environment:

$ conda create --name test_env PACKAGE_NAME

Listing all your conda virtual environments

In case you forget the name of your virtual environments, you can list all your virtual environments by running conda env list

$ conda env list
# conda environments:
#
jupyter_test             ~/.conda/envs/jupyter_test
modeller                 ~/.conda/envs/modeller
psypy3                   ~/.conda/envs/psypy3
test                     ~/.conda/envs/test
test_env                 ~/.conda/envs/test_env
test_pytorch             ~/.conda/envs/test_pytorch
tomopy                   ~/.conda/envs/tomopy
base                  *  /share/apps/rc/software/Anaconda3/5.2.0
DeepNLP                  /share/apps/rc/software/Anaconda3/5.2.0/envs/DeepNLP
ubrite-jupyter-base-1.0     /share/apps/rc/software/Anaconda3/5.2.0/envs/ubrite-jupyter-base-1.0

NOTE: Virtual environment with the asterisk(*) next to it is the one that's currently active.

Activating a conda virtual environment

You can activate your virtual environment for use by running source activate followed by conda activate ENV_NAME

$ source activate
$ conda activate test_env
(test_env) $

NOTE: Your shell prompt would also include the name of the virtual environment that you activated.


IMPORTANT!

The following only applies to versions prior to 2019.10. source activate <env> is not idempotent. Using it twice with the same environment in a given session can lead to unexpected behavior. The recommended workflow is to use source activate to source the conda activate script, followed by conda activate <env>.

From version 2019.10 and on, simply use conda activate <env>.

Locate and install packages

Conda allows you to search for packages that you want to install:

(test_env) $ conda search BeautifulSoup4
Loading channels: done
# Name                  Version           Build  Channel
beautifulsoup4            4.4.0          py27_0  pkgs/free
beautifulsoup4            4.4.0          py34_0  pkgs/free
beautifulsoup4            4.4.0          py35_0  pkgs/free
...
beautifulsoup4            4.6.3          py35_0  pkgs/main
beautifulsoup4            4.6.3          py36_0  pkgs/main
beautifulsoup4            4.6.3          py37_0  pkgs/main
(test_env) $

NOTE: Search is case-insensitive

You can install the packages in conda environment using

(test_env) $ conda install beautifulsoup4
Solving environment: done

## Package Plan ##

  environment location: ~/.conda/envs/test_env

  added / updated specs:
    - beautifulsoup4


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    beautifulsoup4-4.6.3       |           py37_0         138 KB

The following NEW packages will be INSTALLED:

    beautifulsoup4: 4.6.3-py37_0

Proceed ([y]/n)? y


Downloading and Extracting Packages
beautifulsoup4-4.6.3 | 138 KB    | ########################################################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
(test_env) $

Deactivating your virtual environment

You can deactivate your virtual environment using source deactivate

(test_env) $ source deactivate
$

Sharing an environment

You may want to share your environment with someone for testing or other purposes. Sharing the environemnt file for your virtual environment is the most starightforward metohd which allows other person to quickly create an environment identical to you.

Export environment

  • Activate the virtual environment that you want to export.
  • Export an environment.yml file
conda env export -n test_env > environment.yml
  • Now you can send the recently created environment.yml file to the other person.

Create a virtual environment using environment.yml

conda env create -f environment.yml -n test_env

Delete a conda virtual environment

You can use the remove parameter of conda to delete a conda virtual environment that you don't need:

$ conda remove --name test_env --all

Remove all packages in environment ~/.conda/envs/test_env:


## Package Plan ##

  environment location: ~/.conda/envs/test_env


The following packages will be REMOVED:

    beautifulsoup4:  4.6.3-py37_0
    ca-certificates: 2018.03.07-0
    certifi:         2018.8.24-py37_1
    libedit:         3.1.20170329-h6b74fdf_2
    libffi:          3.2.1-hd88cf55_4
    libgcc-ng:       8.2.0-hdf63c60_1
    libstdcxx-ng:    8.2.0-hdf63c60_1
    ncurses:         6.1-hf484d3e_0
    openssl:         1.0.2p-h14c3975_0
    pip:             10.0.1-py37_0
    python:          3.7.0-h6e4f718_3
    readline:        7.0-h7b6447c_5
    setuptools:      40.4.3-py37_0
    sqlite:          3.25.2-h7b6447c_0
    tk:              8.6.8-hbc83047_0
    wheel:           0.32.1-py37_0
    xz:              5.2.4-h14c3975_4
    zlib:            1.2.11-ha838bed_2

Proceed ([y]/n)? y

Moving conda directory

As you build new conda environments, you may find that it is taking a lot of space in your $HOME directory, leading to issues with interactive sessions failing to start. Below are 2 methods to resolve the issue.

Method 1: Move a pre-existing conda directory and create a symlink

cd ~
mv ~/.conda $USER_DATA/
ln -s $USER_DATA/.conda .conda

Method 2: Create a "$HOME/.condarc" file in the $HOME directory by running the following code

cat << "EOF" > ~/.condarc
pkgs_dirs:
  - $USER_DATA/.conda/pkgs
envs_dirs:
  - $USER_DATA/.conda/envs
EOF