Big Data Management

Reproducibility & Collaboration Checklist
Version Control Dependency Management
Back to Beginner's Guide

Dependencies are any external software, libraries, or packages that your research workflow requires to execute. For example, data analysis in Python often relies on the NumPy or Pandas packages, while R workflows might rely heavily on the Tidyverse ecosystem.

As your projects grow, it is critical to keep track of exactly which dependencies—and which specific versions of those dependencies—are being used. Unclear or insufficient dependency management is a leading cause of code breaking over time and is a major barrier to other researchers (or future you) reproducing your results.

This page covers basic procedures, best practices, and tools for managing dependencies and encapsulating your computational environments.

Important: Project Isolation
A common beginner mistake is installing all packages globally into your computer's base environment. You should strive to create a separate, isolated environment for every distinct research project. This prevents version conflicts (e.g., Project A needs Python 3.9 but Project B requires Python 3.11) and makes it much easier to document your work.

1. Virtual Environments

Virtual environments are lightweight, self-contained directories that contain a specific installation of a programming language and its packages. They are the first line of defense for reproducible code on your local machine.

Virtual Environment Introduction
Introduces the core concepts of virtual environments for dependency management. Covers how to set up isolated workspaces using common tools like venv or conda for your analysis scripts.

2. Containerization

While virtual environments handle software packages, they do not capture the underlying operating system or system-level libraries. Containers solve this by packaging your code along with the entire system environment it needs to run, ensuring perfect reproducibility across different machines and servers.

Introduction to containers & Why Use Docker
A conceptual overview of what containers are, how they differ from traditional virtual machines, and why Docker has become an essential tool for robust research workflows.

Docker Basic Tutorial
A hands-on guide to building your first container. Learn how to write a basic Dockerfile, build an image, and execute your code inside an isolated container environment.

Running Docker locally and on Hydras
Specific instructions for scaling up your containerized workflows. Covers how to test your Docker containers locally and subsequently deploy them on the department's Hydra computing clusters.