Ajshen at 03:27, 29 June 2026

2026-06-29T03:27:57Z

← Older revision		Revision as of 03:27, 29 June 2026
Line 1:		Line 1:

			[[Category:Containers and Virtual Environments]]

	Virtual environments are useful tools to manage packages and the associated dependencies. In modern statistical practice, they are also associated with producing reproducible results. Virtual environments also help isolate potentially problematic packages from your “base” environment that can be useful when using certain packages.		Virtual environments are useful tools to manage packages and the associated dependencies. In modern statistical practice, they are also associated with producing reproducible results. Virtual environments also help isolate potentially problematic packages from your “base” environment that can be useful when using certain packages.

Ajshen: replace conda with mamba

2025-02-21T03:25:00Z

replace conda with mamba

Show changes

Skipper: Created page with " Virtual environments are useful tools to manage packages and the associated dependencies. In modern statistical practice, they are also associated with producing reproducible..."

2024-03-29T23:43:36Z

Created page with " Virtual environments are useful tools to manage packages and the associated dependencies. In modern statistical practice, they are also associated with producing reproducible..."

New page

Virtual environments are useful tools to manage packages and the associated dependencies. In modern statistical practice, they are also associated with producing reproducible results. Virtual environments also help isolate potentially problematic packages from your “base” environment that can be useful when using certain packages.

Like everything in life nowadays, you have a few options that can help you make virtual environments. We will mention and go through the basics of both, but strongly recommend using <code>conda</code> and not <code>virtualenv</code>.

The two options are

#<code>conda</code>
#<code>virtualenv</code>

Note that <code>conda</code> is actually a language-agnostic package manager. For example, it can be used to manage both Pyhon and R within the same virtual environment. Interested in R-specific virtual environments? Check [[Renv|virtual environments for R]].

= A note on installing packages =

There are two main ways to install python packages, either through <code>pip</code> or <code>conda</code>.
<syntaxhighlight lang="bash">pip install scikit-learn</syntaxhighlight>

or
<syntaxhighlight lang="bash">conda install scikit-learn</syntaxhighlight>

<code>pip</code> will provide you with the most up-to-date package and <code>conda</code> only caries a subset of all of the packages avaiable on <code>pip</code> (aka from <code>pypi</code>). A major difference between <code>pip</code> installation of packages and <code>conda</code> installation is that <code>pip</code> doesn’t check the dependence struture of packages (this is required to be done manually), whereas <code>conda</code> makes sure no dependences are broken. This can become very useful especially if you don’t love constantly debugging. If a package cannot be obtained with <code>conda</code>, <code>conda</code> will still track packages you install using <code>pip install</code>.

N.B. Altough this is a nice <code>conda</code> functionality, it can behave annoingly sometimes. For example, conda retrieves packages from different “repositories” called channels (<code>-c</code> option when running <code>conda install</code>). Sometimes using different channels can yield different results and download slightly different versions of the same package, which easily causes inconsistencies in the environment. We strongly suggest to use the same channel within an environment, and only change it if a package is only available from a specific channel.

If a python package is just on a GitHub repository and not on <code>pypi</code> or <code>conda</code>, then you can download/clone the repository, navigate to the repository and then run
<syntaxhighlight lang="bash">pip install .</syntaxhighlight>

By adding the option <code>-e</code> to the command above, the cloned repository is installed in editable mode, which allows you to modify that code locally and immediately use the changes by importing the repository in your code as an installed package. This is particularly useful when you are actively developing a package locally.

= <code>conda</code> and virtual environments =

As mentioned above, <code>conda</code> should really be thought of as a general package manager, but can also be used to create virtual environments.

== Installing Conda ==

=== <code>conda</code> for your personal computer ===

To install <code>conda</code> on your personal computer you can install anaconda or miniconda with an installer (works for Mac, Windows and Linux) or via command line. Anaconda takes up more space and comes with a decent amount of packages installed, whereas miniconda takes up much less space but doesn't come with packages installed (see [https://conda.io/projects/conda/en/latest/user-guide/install/download.html#anaconda-or-miniconda conda.io] for more commentary on the differences. For either installation, check the hash (a unique identifier of the object) to make sure you're not installing a bug (see [https://conda.io/projects/conda/en/latest/user-guide/install/download.html#id5 | conda.io] for how to check the hash on different OS). Generally speaking, Anaconda can be useful if you're new to programming. Miniconda has all the package-manager functionalities but requires you to install every single package in the environments, which is generally better if you want to manage your projects from scratch.

To install miniconda head here: [https://docs.conda.io/en/latest/miniconda.html docs.conda] and anaconda head here: [https://docs.anaconda.com/anaconda/install/index.html docs.anaconda].

=== <code>conda</code> for your server ===

To install <code>conda</code> on the server you need to use the command line all the way through (you cannot download anything from a webpage. As such, I recommend installing miniconda following these steps (for Linux systems):

#<pre>mkdir -p ~/miniconda3</pre>

<ol start="2">
<li><pre>wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
</pre>
</li>
<li><pre>bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
</pre>
</li>
<li><pre>rm -rf ~/miniconda3/miniconda.sh</pre>
</li>
<li><pre>~/miniconda3/bin/conda init bash</pre>
</li>
</ol>

Similar steps can be followed to install miniconda from the terminal on Mac and Windows systems. See this [https://engineeringfordatascience.com/posts/install_miniconda_from_the_command_line/ link].<br/> '''N.B. for Hydras.''' For reasons related to how Hydras directory system is setup, you will need an extra step to make sure miniconda is initialized and available every time you log into the server. The steps above populate your <code>.bashrc</code> file with a small script that initialized conda at log in, but this file is not sourced by Hydras. Instead, you need to create a <code>.profile</code> file (if it's not already there) and add the following line to it:
<pre>source .bashrc</pre>

== Conda’s virtual environments ==

A good tutorial on how to create a virtual environment with <code>conda</code> can be found at [https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#creating-an-environment-with-commands conda-io]

In a nutshell:

#To a '''create new environment''' with Python installed (note that <code>base</code> is your standard environment) you should run: <syntaxhighlight lang="bash">conda create -n yourenvname python=x.x</syntaxhighlight>

           where <code>x.x</code> is the python version you’d like to use. I might suggest starting with 3.11 now.
<ol start="2">
<li>Then to '''activate your conda environment''', just run: <syntaxhighlight lang="bash">conda activate yourenvname</syntaxhighlight>
</li>
</ol>

           This will probably make your terminal show <code>(yourenvname) ...$</code>, whereas originally it should have read <code>(base) ...$</code>.
<ol start="3">
<li>To '''install packages''' in your local environment:<br/> After activating (and before deactivating your virtual environment) you can just do <syntaxhighlight lang="bash">conda activate yourenvname
conda install [package]</syntaxhighlight>
</li>
</ol>

           An alternative to the second line is  <code>pip install [package]</code>
<ol start="4">
<li>To '''deactivate''' your environment: <syntaxhighlight lang="bash">conda deactivate</syntaxhighlight>
</li>
</ol>
<ol start="5">
<li>To '''share''' virtual environments (useful for reproducibility):<br/> To share your virtual environment (with collaborators or yourself on the server) you can make a <code>.yml</code> file that contains all the information about the packages you are using by first opening up the virtual environment and then running <syntaxhighlight lang="bash"> conda env export > environment.yml</syntaxhighlight>
</li>
</ol>

           Then you can share the <code>environment.yml</code> file to another computer, and with that computer create a copy of the environment via   <code data-lang="bash">conda env create -f environment.yml</code>

More guidance can be found here: [https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#sharing-an-environment conda.io]

= <code>virtualenv</code> virtual environments =

If you’re only using <code>pip</code> as your package installer you can also get virtual environments using <code>virtualenv</code>. These environments require you to specify a specific location for their information to be stored (so often are project dependent). Some good references are [https://docs.python-guide.org/dev/virtualenvs/ docs.python-guide] and [https://realpython.com/python-virtual-environments-a-primer/ realpython].

To start you should download <code>virtualenv</code> with
<syntaxhighlight lang="bash">pip install virtualenv</syntaxhighlight>

Then to create a new virtual environment and use it, you can follow these basic steps:

#You should select where you’d like to store the information about the virtual environment.
#To '''create''' a new virtual environment, in that folder you can run (right in the terminal): <syntaxhighlight lang="bash">virtualenv yourenvname</syntaxhighlight>

<ol start="3">
<li>To '''activate''' run: <syntaxhighlight lang="bash">$ source yourenvname/bin/activate</syntaxhighlight>
</li>
</ol>

           This will probably make your terminal show  <code>(yourenvname) ...$</code>
<ol start="4">
<li>To '''deactivate''': <syntaxhighlight lang="bash">(env) ... $ deactivate
... $</syntaxhighlight>
</li>
</ol>

Python virtual environments - Revision history

Ajshen at 03:27, 29 June 2026

Ajshen: replace conda with mamba

Skipper: Created page with " Virtual environments are useful tools to manage packages and the associated dependencies. In modern statistical practice, they are also associated with producing reproducible..."