Big Data Management

Scaling Checklist
Building Workflows on Remote Servers Parallelization: Making the most of your Machine Big Data Management
Back to Beginner's Guide

As your research experiments scale up in size, effective data management and storage become just as important as computational efficiency. Working with large datasets—ranging from hundreds of gigabytes to several terabytes—requires specialized tools to move, store, and read/write data efficiently without bottlenecking your workflow or filling up the cluster.

This page covers basic procedures, best practices, and tools for handling large amounts of data on remote servers.

Important: Storage Quotas and Scratch Space
Unlike your personal laptop, department and shared servers usually have strict storage quotas in your personal `/home` directory. It is therefore important to budget available space accordingly and adhere to best practices (such as regularly cleanup of temporary files).

1. Moving Data

Transferring gigabytes or terabytes of data over standard internet connections can be slow and prone to interruption. The tools below are designed to move data reliably between your local machine, the department servers, and external clusters.

Command Line Data Transfer
Introduces basic Linux tools like scp, rsync, and wget. Ideal for moving small to medium-sized datasets or downloading data directly from the web to the server.

Globus
A graphical, web-based application for moving and sharing massive files within and between institutions. Globus handles network interruptions automatically and is highly recommended for terabyte-scale transfers.

Rclone (TBW)
A command-line tool that acts like rsync for cloud storage. Essential if your dataset is currently hosted on Google Drive, AWS S3, or Dropbox and needs to be pulled down to the cluster.

2. File Formats and I/O

When dealing with large data, traditional text formats like .csv take too long to read into memory and take up too much disk space. Transitioning to binary or columnar formats is a critical step in scaling up your research.

Efficient Data Formats
An overview of high-performance file formats like Parquet, Feather, and HDF5. Explains how to read and write these formats in R and Python to drastically speed up your load times.

Data Compression on Linux
How to use tar, gzip, and zip on the command line to compress large directories of files before transferring or archiving them.

3. Out-of-Memory Processing

Sometimes, a dataset is simply too large to load into the RAM of even our largest servers. These tools allow you to query and manipulate datasets that exceed available memory.

Out-of-Memory Data Manipulation (TBW)
An introduction to modern tools like DuckDB, Apache Arrow, or Dask. Learn how to filter, mutate, and aggregate massive datasets on disk before loading the smaller, processed results into RAM for statistical modeling.