View source for Big Data Management

[[Category:Beginner's_Guide_Scaling|03]]

{{ScalingSidebar}}

'''Dependencies''' are any external software, libraries, or packages that your research workflow requires to execute. For example, data analysis in Python often relies on the NumPy or Pandas packages, while R workflows might rely heavily on the Tidyverse ecosystem.

As your projects grow, it is critical to keep track of exactly which dependencies—and '''which specific versions''' of those dependencies—are being used. Unclear or insufficient dependency management is a leading cause of code breaking over time and is a major barrier to other researchers (or future you) reproducing your results.

This page covers basic procedures, best practices, and tools for managing dependencies and encapsulating your computational environments.

<div style="background-color: #fff3cd; border-left: 5px solid #ffc107; padding: 15px; margin-bottom: 20px; margin-top: 15px; overflow: hidden;">
'''Important: Project Isolation'''<br>
A common beginner mistake is installing all packages globally into your computer's base environment. You should strive to create a '''separate, isolated environment''' for every distinct research project. This prevents version conflicts (e.g., Project A needs Python 3.9 but Project B requires Python 3.11) and makes it much easier to document your work.
</div>

== 1. Virtual Environments ==
Virtual environments are lightweight, self-contained directories that contain a specific installation of a programming language and its packages. They are the first line of defense for reproducible code on your local machine.

<div style="background-color: #f8f9fa; border-left: 4px solid #6c757d; padding: 15px; margin-bottom: 10px;">
'''[[Virtual Environment Introduction]]'''<br>
Introduces the core concepts of virtual environments for dependency management. Covers how to set up isolated workspaces using common tools like <code>venv</code> or <code>conda</code> for your analysis scripts.
</div>

== 2. Containerization ==
While virtual environments handle software packages, they do not capture the underlying operating system or system-level libraries. Containers solve this by packaging your code along with the ''entire'' system environment it needs to run, ensuring perfect reproducibility across different machines and servers.

<div style="background-color: #f8f9fa; border-left: 4px solid #005A9C; padding: 15px; margin-bottom: 10px;">
'''[[Introduction to containers]] & [[Docker, and why-you-should-use-it-for-your-research|Why Use Docker]]'''<br>
A conceptual overview of what containers are, how they differ from traditional virtual machines, and why Docker has become an essential tool for robust research workflows.
</div>

<div style="background-color: #f8f9fa; border-left: 4px solid #75AADB; padding: 15px; margin-bottom: 10px;">
'''[[Tutorials|Docker Basic Tutorial]]'''<br>
A hands-on guide to building your first container. Learn how to write a basic <code>Dockerfile</code>, build an image, and execute your code inside an isolated container environment.
</div>

<div style="background-color: #f8f9fa; border-left: 4px solid #6f42c1; padding: 15px; margin-bottom: 10px;">
'''[[Running Docker locally and on Hydras]]'''<br>
Specific instructions for scaling up your containerized workflows. Covers how to test your Docker containers locally and subsequently deploy them on the department's Hydra computing clusters.
</div>