Python virtual environments

Virtual environments are useful tools to manage packages and the associated dependencies. In modern statistical practice, they are also associated with producing reproducible results. Virtual environments also help isolate potentially problematic packages from your “base” environment that can be useful when using certain packages.

Like everything in life nowadays, you have a few options that can help you make virtual environments. We will mention and go through the basics of each, but strongly recommend using mamba and not virtualenv.

The two options are

1. mamba
2. virtualenv

Note that mamba is actually a language-agnostic package manager. For example, it can be used to manage both Python and R within the same virtual environment. Interested in R-specific virtual environments? Check virtual environments for R.

(conda is no longer recommended as a virtual environment manager due to licensing issues. Instructions for setting up conda is still available here for reference)

A note on installing packages

There are two main ways to install python packages, either through pip or mamba.

pip install scikit-learn

or

mamba install scikit-learn

pip will provide you with the most up-to-date package and mamba only caries a subset of all of the packages avaiable on pip (aka from pypi). A major difference between pip installation of packages and mamba installation is that pip doesn’t enforce dependencies (as of pip 20.3 released in November 2020, it will give a warning when dependencies are broken) whereas mamba makes sure no dependences are broken. This can become very useful especially if you don’t love constantly debugging. If a package cannot be obtained with mamba, mamba will still track packages you install using pip install.

Although this is a nice mamba functionality, it can behave annoying sometimes. For example, mamba retrieves packages from different “repositories” called channels (-c option when running mamba install). Sometimes using different channels can yield different results and download slightly different versions of the same package, which easily causes inconsistencies in the environment. We strongly suggest to use the same channel within an environment, and only change it if a package is only available from a specific channel.

If a python package is just on a GitHub repository and not on pypi or mamba, then you can download/clone the repository, navigate to the repository and then run

pip install .

By adding the option -e to the command above, the cloned repository is installed in editable mode, which allows you to modify that code locally and immediately use the changes by importing the repository in your code as an installed package. This is particularly useful when you are actively developing a package locally.

Installing Mamba

mamba for your personal computer

To install mamba on your personal computer you can install "Mamba" or "Micromamba" with an installer (works for Mac, Windows and Linux) or via command line. For must use cases, "Mamba" will be sufficient, but see here for a description of Micromamba and how it differs from "Mamba". To install Mamba head here, and for Micromamba head here.

mamba for your server

To install mamba on the server you need to use the command line all the way through (you cannot download anything from a webpage. As such, I recommend installing mamba following these steps (for Linux systems):

1. mkdir -p ~/mamba

2. wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh -O ~/mamba/install.sh
   

3. bash ~/mamba/install.sh -b -u -p ~/mamba
   

4. rm -rf ~/mamba/install.sh

5. ~/mamba/bin/conda init bash

Similar steps can be followed to install minimamba from the terminal on Mac and Windows systems.
Note for Hydra Users: For reasons related to how Hydras directory system is setup, you will need an extra step to make sure mamba is initialized and available every time you log into the server. The steps above populate your .bashrc file with a small script that initialized mamba at log in, but this file is not sourced by Hydras. Instead, you need to create a .profile file (if it's not already there) and add the following line to it:

source .bashrc

Mamba's virtual environments

A good tutorial on how to create a virtual environment with mamba can be found at here

In a nutshell:

1. To a create new environment with Python installed (note that base is your standard environment) you should run:

   mamba create -n yourenvname python=x.x

           where x.x is the python version you’d like to use. I might suggest starting with 3.11 now.

2. Then to activate your conda environment, just run:

   mamba activate yourenvname

           This will probably make your terminal show (yourenvname) ...$, whereas originally it should have read (base) ...$.

3. To install packages in your local environment:
   After activating (and before deactivating your virtual environment) you can just do

   mamba activate yourenvname
   mamba install [package]

           An alternative to the second line is  pip install [package]

4. To deactivate your environment:

   mamba deactivate

5. To share virtual environments (useful for reproducibility):
   To share your virtual environment (with collaborators or yourself on the server) you can make a .yml file that contains all the information about the packages you are using by first opening up the virtual environment and then running

    mamba env export > environment.yml

           Then you can share the environment.yml file to another computer, and with that computer create a copy of the environment via   mamba env create -f environment.yml

virtualenv virtual environments

If you’re only using pip as your package installer you can also get virtual environments using virtualenv. These environments require you to specify a specific location for their information to be stored (so often are project dependent). Some good references are docs.python-guide and realpython.

To start you should download virtualenv with

pip install virtualenv

Then to create a new virtual environment and use it, you can follow these basic steps:

1. You should select where you’d like to store the information about the virtual environment.
2. To create a new virtual environment, in that folder you can run (right in the terminal):

   virtualenv yourenvname

3. To activate run:

   $ source yourenvname/bin/activate

           This will probably make your terminal show  (yourenvname) ...$

4. To deactivate:

   (env) ... $ deactivate
   ... $