from Anaconda, Inc
is a completely free enterprise-ready distribution for large-scale data
processing, predictive analytics, and scientific computing. It includes
over 195 of the most popular Python packages for science, math,
engineering, and data analysis. It also offers the ability to easily
create custom environments by mixing and matching different versions
of Python and/or R and other packages into isolated environments that
individual users are free to create. Anaconda includes the
package and environment manager to make managing these environments
While the standard methods of installing packages via
easy_install work with Anaconda, the preferred method is using
Full documentation on using Conda is available at http://conda.pydata.org/docs/
A cheatsheet is also provided.
A few examples of the basic commands are provided here. For a full explanation of all of Anaconda/Conda’s capabilities, see the documentation linked above.
Anaconda is provided through the
anaconda module on HCC machines. To
begin using it, load the Anaconda module.
module load anaconda
To display general information about Conda/Anaconda, use the
Conda allows the easy creation of isolated, custom environments with
packages and versions of your choosing. To show all currently available
environments, and which is active, use the
infosubcommand with the
conda info -e
The active environment will be marked with an asterisk (*) character.
list command will show all packages installed
in the currently active environment.
To find packages, use the
conda search numpy
If the package is available, this will also display available package versions and compatible Python versions the package may be installed under.
create command is used to create a new environment. It requires
at a minimum a name for the environment, and at least one package to
install. For example, suppose we wish to create a new environment, and
need version 1.17 of NumPy.
conda create -n mynumpy numpy=1.17
This will create a new environment called ‘mynumpy’ and installed NumPy version 1.17, along with any required dependencies.
To use the environment, we must first activate it.
conda activate mynumpy
Our new environment is now active, and we can use it. The shell prompt will change to indicate this as well.
By default, conda environments are installed in the user’s
home directory at
This is fine for smaller environments, but larger environments (especially ML/AI-based ones) can quickly
exhaust the space in the
For larger environments, we recommend using the
$COMMON folder instead. To do so, use the
conda create. For example, creating the same environment as above but
placing it in the folder
conda create -p $COMMON/mynumpy numpy=1.17
To activate the environment, you must use the full path.
conda activate $COMMON/mynumpy
Please note that you’ll need to add the
#SBATCH --licenses=common directive to your submit scripts
as described here in order to use environments
To install additional packages in an environment, use the
subcommand. Suppose we want to install iPython in our ‘mynumpy’
environment. While the environment is active, use
conda install ipython
If you aren’t currently in the environment you wish to install the
package in, add the
-noption to specify the name.
conda install -n mynumpy ipython
remove subcommand to uninstall a package functions similarly.
conda remove ipython
conda remove -n mynumpy ipython
To exit an environment, we deactivate it.
Finally, to completely remove an environment, add the
conda remove -n mynumpy --all
We provide GPU versions of various frameworks such as
theano, via modules.
However, sometimes you may need additional libraries or packages that are not available as part of these modules.
In this case, you will need to create your own GPU Anaconda environment.
To do this, you need to first clone one of our GPU modules to a new Anaconda environment, and then install the desired packages in this new environment.
The reason for this is that the GPU modules we support are built using the specific CUDA drivers our GPU nodes have. If you just create custom GPU environment without cloning the module, your code will not utilize the GPUs correctly.
For example, if you want to use
tensorflow with additional packages, first do:
module load tensorflow-gpu/py39/2.9 module load anaconda conda create -n tensorflow-gpu-2.9-custom --clone $CONDA_DEFAULT_ENV module purge
This will create a new
tensorflow-gpu-2.9-custom environment in your home directory that is a copy of the
Then, you can install the additional packages you need in this environment.
module load anaconda conda activate tensorflow-gpu-2.9-custom conda install <packages>
Next, whenever you want to use this custom GPU Anaconda environment, you need to add these two lines in your submit script:
module load anaconda conda activate tensorflow-gpu-2.9-custom
If you have custom GPU Anaconda environment please only use the two lines from above and DO NOT load the module you have cloned earlier.
module load tensorflow-gpu/py39/2.9 and
conda activate tensorflow-gpu-2.9-custom in the same script is wrong and may give you various errors and incorrect results.
tensorflow-gpu/py39/2.9 is used here as an example module and version, please make sure you use the newest available version of the module you want to clone, or the version that is needed for your particular research needs.
Some conda packages available on
bioconda support MPI (via
mpich). However, just using the
mpich packages from
conda-forge often does not work on HPC systems. More information about this can be found here.
In order to be able to correctly use these MPI packages with the MPI libraries installed on our clusters, two steps need to be performed.
First, at install time, besides the package, the “dummy” package
mpich=4.0.*=external_* needs to be installed for
These “dummy” packages are empty, but allow the solver to create correct environments and use the system-wide modules when the environment is activated.
Secondly, when activating the conda environment and using the package, the system-wide
mpich/4.0 module needs to be loaded depending on the MPI library used.
Currently only packages that were built using
openmpi 4.1 and
mpich 4.0 are supported on HCC clusters.
For example, the steps for creating conda environment with
mpi4py that supports
module purge module load anaconda conda create -n mpi4py-openmpi mpi4py openmpi=4.1.*=external_*
module purge module load compiler/gcc/10 openmpi/4.1 anaconda conda activate mpi4py-openmpi
The steps for creating conda environment with
mpi4py that supports
module purge module load anaconda conda create -n mpi4py-mpich mpi4py mpich=4.0.*=external_*
module purge module load compiler/gcc/10 mpich/4.0 anaconda conda activate mpi4py-mpich
It is not difficult to make an Anaconda environment available to a
Jupyter Notebook. To do so, follow the steps below, replacing
myenv with the name of the Python or R environment you wish to use:
Stop any running Jupyter Notebooks and ensure you are logged out of the JupyterHub instance on the cluster you are using.
Using the command-line environment of the login node, load the target conda environment:
conda activate myenv
Install the Jupyter kernel and add the environment:
For a Python conda environment, install the IPykernel package, and then the kernel specification:
# Install ipykernel conda install ipykernel # Install the kernel specification python -m ipykernel install --user --name "$CONDA_DEFAULT_ENV" --display-name "Python ($CONDA_DEFAULT_ENV)" --env PATH $PATH
If needed, other variables can be set via additional `--env` arguments, e.g., `python -m ipykernel install --user --name "$CONDA_DEFAULT_ENV" --display-name "Python ($CONDA_DEFAULT_ENV)" --env PATH $PATH --env VAR value`, where `VAR` and `value` are the name and the value of the variable respectively.
For an R conda environment, install the jupyter_client and IRkernel packages, and then the kernel specification:
# Install PNG support for R, the R kernel for Jupyter, and the Jupyter client conda install r-png conda install r-irkernel jupyter_client # Install jupyter_client 5.2.3 from anaconda channel for bug workaround conda install -c anaconda jupyter_client # Install the kernel specification R -e "IRkernel::installspec(name = '$CONDA_DEFAULT_ENV', displayname = 'R ($CONDA_DEFAULT_ENV)', user = TRUE)"
Once you have the environment set up, deactivate it:
Login to JupyterHub
and create a new notebook using the environment by selecting the
correct entry in the
New dropdown menu in the top right