Command Line Data Transfer

When transitioning to remote server workflows, your first instinct might be to drag and drop files using the built-in file explorer in VS Code or other IDEs. While fine for small scripts, GUI-based transfers quickly break down when moving gigabytes or terabytes of research data.

This guide covers the essential command-line tools for reliably moving data to and from the department servers.

1. The Limits of GUI-Based Transfers

Integrated Development Environments (IDEs) like VS Code Remote are incredible for writing code, but their built-in file uploaders are not built for big data. Relying on them for large datasets introduces several risks:

• No Checkpointing: If your Wi-Fi drops 90% of the way through a 50GB transfer, GUI tools usually force you to start over from the beginning.
• UI Freezing: Moving thousands of small files (like image datasets) can overwhelm the IDE's file watcher, causing your entire editor to freeze or crash.
• Memory Overhead: GUI tools often load chunks of the file into your local and remote RAM during transfer, which is incredibly inefficient and can trigger out-of-memory errors.

Command-line interface (CLI) tools bypass the visual overhead, offer robust error handling, and allow you to pause and resume massive transfers.

2. Essential CLI Transfer Tools

Below are the three core commands you will use for data management.

Prerequisite: For tools like scp and rsync, you must run these commands from your local machine's terminal, not from inside the server.

wget: Downloading Directly to the Server

If your data is already hosted online (e.g., on GitHub, an open data portal, or a public AWS bucket), do not download it to your laptop first. Use wget to pull it directly onto the server at gigabit speeds.

Step-by-step Example:

1. Connect to the server: ssh your_stat_id@leviathan1.stat.cmu.edu
2. Navigate to your target folder: cd /scratch/your_andrew_id/data
3. Download the file:

wget https://example-data-portal.com/massive_dataset.csv

scp (Secure Copy): Best for Single Files

scp uses the same security protocol as SSH. It is the easiest way to push a specific file from your laptop to the server, or pull a results file back down.

Step-by-step Example:

1. Open a terminal on your local computer.
2. To push a file to the server:

scp ./local_model_weights.pt your_stat_id@leviathan1.stat.cmu.edu:/home/your_stat_id/models/

1. To pull a file from the server:

scp your_stat_id@leviathan1.stat.cmu.edu:/home/your_stat_id/results.csv ./Desktop/

rsync: The Gold Standard for Big Data

If you are moving entire directories or datasets, rsync is mandatory. It checks what files already exist at the destination and only transfers the differences. If your internet dies, simply run the exact same rsync command again, and it will pick up right where it left off.

Step-by-step Example:

1. Open a terminal on your local computer.
2. Sync a local folder to the server. The flags -avP stand for Archive (keeps file structures intact), Verbose (shows progress), and Partial (allows resuming).

rsync -avP ./my_massive_dataset/ your_stat_id@leviathan1.stat.cmu.edu:/scratch/your_stat_id/dataset/

See 🔗 https://www.geeksforgeeks.org/linux-unix/rsync-command-in-linux-with-examples/

3. Where to Put Your Data: Home vs. Scratch

Before you start moving terabytes of data, it is critical to understand where you are putting it. The department clusters have two distinct types of storage:

/home (Network Attached Storage)

Your home directory (e.g., /home/your_andrew_id) is hosted on a central Network Attached Storage (NAS) drive.

• The Benefit: Data stored here "follows" you. If you download a file to /home while logged into leviathan1, that exact same file will be waiting for you if you log into leviathan2 or the ghidorah nodes.
• The Drawback: NAS drives have strict storage quotas and slower read/write speeds over the network. Do not put massive datasets here.

/scratch (Local Disk)

Every individual server node has its own dedicated physical hard drive, usually mounted at /scratch.

• The Benefit: Massive capacity and extremely fast disk I/O, meaning your code will read the data much faster. Always transfer your large datasets here.
• The Drawback: Scratch space is local to the machine. If you download a dataset to /scratch/your_andrew_id/ on leviathan1, you cannot see or access it from leviathan2.

Best Practice: Keep your code, scripts, and small configuration files in /home so you can run them anywhere. Keep your large .csv, .parquet, or image files in /scratch on the specific node you are computing on.

4. Special Note: Slurm-Managed Clusters

If you are graduating from the Leviathans to massive shared clusters managed by Slurm (like the Wright GPU cluster or PSC), there is one strict rule regarding data transfer:

Never run heavy data transfers on the login nodes.

When you SSH into a Slurm cluster, you land on a shared "login node." If you run a massive wget or rsync command there, you will consume the network bandwidth for everyone else trying to log in, and system administrators may kill your process.

Instead, you must either:

1. Start an interactive compute session (salloc or srun) and run your wget/rsync command from inside that active compute node.
2. Check the cluster's documentation for a dedicated Data Transfer Node (DTN), which is a specific server address built exclusively to handle heavy rsync or Globus traffic.