View source for Python virtual environments


[[Category:Containers and Virtual Environments]]

Virtual environments are useful tools to manage packages and the associated dependencies. In modern statistical practice, they are also associated with producing reproducible results. Virtual environments also help isolate potentially problematic packages from your “base” environment that can be useful when using certain packages.

Like everything in life nowadays, you have a few options that can help you make virtual environments. We will mention and go through the basics of each, but strongly recommend using <code>mamba</code> and not <code>virtualenv</code>.

The two options are

#<code>mamba</code>
#<code>virtualenv</code>

Note that&nbsp;<code>mamba</code> is actually a language-agnostic package manager. For example, it can be used to manage both Python and R within the same virtual environment. Interested in R-specific virtual environments? Check [[Renv|virtual environments for R]].

(<code>conda</code> is no longer recommended as a virtual environment manager due to licensing issues. Instructions for setting up <code>conda</code> is still available [[Conda with Python (Reference Only)|here]] for reference)

=A note on installing packages=

There are two main ways to install python packages, either through <code>pip</code> or <code>mamba</code>.
<syntaxhighlight lang="bash">pip install scikit-learn</syntaxhighlight>

or
<syntaxhighlight lang="bash">mamba install scikit-learn</syntaxhighlight>

<code>pip</code> will provide you with the most up-to-date package and <code>mamba</code> only caries a subset of all of the packages avaiable on <code>pip</code> (aka from <code>pypi</code>). A major difference between <code>pip</code> installation of packages and <code>mamba</code> installation is that <code>pip</code> doesn’t enforce dependencies (as of pip 20.3 released in November 2020, it will give a warning when dependencies are broken) whereas <code>mamba</code> makes sure no dependences are broken. This can become very useful especially if you don’t love constantly debugging. If a package cannot be obtained with <code>mamba</code>, <code>mamba</code> will still track packages you install using <code>pip install</code>.

Although this is a nice <code>mamba</code> functionality, it can behave annoying sometimes. For example, <code>mamba</code> retrieves packages from different “repositories” called channels (<code>-c</code> option when running <code>mamba install</code>). Sometimes using different channels can yield different results and download slightly different versions of the same package, which easily causes inconsistencies in the environment. We strongly suggest to use the same channel within an environment, and only change it if a package is only available from a specific channel.

If a python package is just on a GitHub repository and not on <code>pypi</code> or <code>mamba</code>, then you can download/clone the repository, navigate to the repository and then run
<syntaxhighlight lang="bash">pip install .</syntaxhighlight>

By adding the option&nbsp;<code>-e</code> to the command above, the cloned repository is installed in editable mode, which allows you to modify that code locally and immediately use the changes by importing the repository in your code as an installed package. This is particularly useful when you are actively developing a package locally.

==Installing Mamba ==

===<code>mamba</code> for your personal computer===

To install <code>mamba</code> on your personal computer you can install "Mamba" or "Micromamba" with an installer (works for Mac, Windows and Linux) or via command line. For must use cases, "Mamba" will be sufficient, but see here for a [https://mamba.readthedocs.io/en/latest/user_guide/micromamba.html#micromamba description of Micromamba] and how it differs from "Mamba". To install Mamba head [https://mamba.readthedocs.io/en/latest/installation/mamba-installation.html here], and for Micromamba head [https://mamba.readthedocs.io/en/latest/installation/micromamba-installation.html here].

===<code>mamba</code> for your server ===

To install <code>mamba</code> on the server you need to use the command line all the way through (you cannot download anything from a webpage. As such, I recommend installing mamba following these steps (for Linux systems):

#<pre>mkdir -p ~/mamba</pre>

<ol start="2">
<li><pre>wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh -O ~/mamba/install.sh
</pre>
</li>
<li><pre>bash ~/mamba/install.sh -b -u -p ~/mamba
</pre>
</li>
<li><pre>rm -rf ~/mamba/install.sh</pre>
</li>
<li><pre>~/mamba/bin/conda init bash</pre>
</li>
</ol>

Similar steps can be followed to install minimamba from the terminal on Mac and Windows systems. <br />'''Note for Hydra Users:'''&nbsp;For reasons related to how Hydras directory system is setup, you will need an extra step to make sure mamba is initialized and available every time you log into the server. The steps above populate your&nbsp;<code>.bashrc</code> file with a small script that initialized mamba at log in, but this file is not sourced by Hydras. Instead, you need to create a&nbsp;<code>.profile</code> file (if it's not already there) and add&nbsp;the following line to it:
<pre>source .bashrc</pre>


==Mamba's virtual environments==

A good tutorial on how to create a virtual environment with <code>mamba</code> can be found at [https://mamba.readthedocs.io/en/latest/user_guide/mamba.html here]

In a nutshell:

#To a&nbsp;'''create new environment'''&nbsp;with Python installed (note that <code>base</code> is your standard environment) you should run: <syntaxhighlight lang="bash">mamba create -n yourenvname python=x.x</syntaxhighlight>


&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;where <code>x.x</code> is the python version you’d like to use. I might suggest starting with 3.11&nbsp;now.
<ol start="2">
<li>Then to '''activate your conda environment''', just run: <syntaxhighlight lang="bash">mamba activate yourenvname</syntaxhighlight>
</li>
</ol>

&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;This will probably make your terminal show <code>(yourenvname) ...$</code>, whereas&nbsp;originally it should have read <code>(base) ...$</code>.
<ol start="3">
<li>To '''install packages''' in your local environment:<br /> After activating (and before deactivating your virtual environment) you can just do <syntaxhighlight lang="bash">mamba activate yourenvname
mamba install [package]</syntaxhighlight>
</li>
</ol>

&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;An alternative to the second line is&nbsp;&nbsp;<code>pip install [package]</code>
<ol start="4">
<li>To '''deactivate''' your environment: <syntaxhighlight lang="bash">mamba deactivate</syntaxhighlight>
</li>
</ol>
<ol start="5">
<li>To '''share''' virtual environments (useful for reproducibility):<br /> To share your virtual environment (with collaborators or yourself on the server) you can make a <code>.yml</code> file that contains all the information about the packages you are using by first opening up the virtual environment and then running <syntaxhighlight lang="bash"> mamba env export > environment.yml</syntaxhighlight>
</li>
</ol>

&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;Then you can share the <code>environment.yml</code> file to another computer, and with that computer create a copy of the environment via&nbsp; &nbsp;<code data-lang="bash">mamba env create -f environment.yml</code>

=<code>virtualenv</code> virtual environments=

If you’re only using <code>pip</code> as your package installer you can also get virtual environments using <code>virtualenv</code>. These environments require you to specify a specific location for their information to be stored (so often are project dependent). Some good references are [https://docs.python-guide.org/dev/virtualenvs/ docs.python-guide] and [https://realpython.com/python-virtual-environments-a-primer/ realpython].

To start you should download <code>virtualenv</code> with
<syntaxhighlight lang="bash">pip install virtualenv</syntaxhighlight>

Then to create a new virtual environment and use it, you can follow these basic steps:

#You should select where you’d like to store the information about the virtual environment.
#To '''create''' a new virtual environment, in that folder you can run (right in the terminal): <syntaxhighlight lang="bash">virtualenv yourenvname</syntaxhighlight>

<ol start="3">
<li>To '''activate''' run: <syntaxhighlight lang="bash">$ source yourenvname/bin/activate</syntaxhighlight>
</li>
</ol>

&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;This will probably make your terminal show&nbsp;&nbsp;<code>(yourenvname) ...$</code>
<ol start="4">
<li>To '''deactivate''': <syntaxhighlight lang="bash">(env) ... $ deactivate
... $</syntaxhighlight>
</li>
</ol>