Ajshen at 02:54, 29 June 2026

2026-06-29T02:54:14Z

← Older revision		Revision as of 02:54, 29 June 2026
Line 1:		Line 1:
			[[Category:Terminal and Command Line]]

	When transitioning to remote server workflows, your first instinct might be to drag and drop files using the built-in file explorer in VS Code or other IDEs. While fine for small scripts, GUI-based transfers quickly break down when moving gigabytes or terabytes of research data.		When transitioning to remote server workflows, your first instinct might be to drag and drop files using the built-in file explorer in VS Code or other IDEs. While fine for small scripts, GUI-based transfers quickly break down when moving gigabytes or terabytes of research data.

Ajshen: /* 3. Where to Put Your Data: Home vs. Scratch */

2026-06-27T06:18:39Z

3. Where to Put Your Data: Home vs. Scratch

← Older revision		Revision as of 06:18, 27 June 2026
Line 68:		Line 68:

	<blockquote>'''Best Practice:''' Keep your code, scripts, and small configuration files in <code>/home</code> so you can run them anywhere. Keep your large <code>.csv</code>, <code>.parquet</code>, or image files in <code>/scratch</code> on the specific node you are computing on.</blockquote>		<blockquote>'''Best Practice:''' Keep your code, scripts, and small configuration files in <code>/home</code> so you can run them anywhere. Keep your large <code>.csv</code>, <code>.parquet</code>, or image files in <code>/scratch</code> on the specific node you are computing on.</blockquote>

			=== How to Find the Scratch Directory ===
			While local storage is often predictably located at <code>/scratch</code>, the exact path can vary depending on how the server was configured. Here is how to locate it step-by-step:

			# '''Log into the server:''' Open your terminal and connect to your target node.
			# '''Check available disk space:''' Run the <code>df</code> (disk free) command with the <code>-h</code> (human-readable) flag to list all mounted drives and their capacities:
			<syntaxhighlight lang="bash">
			df -h
			</syntaxhighlight>
			# '''Identify the large local drive:''' Look at the output table. You are searching for a line where the "Size" is massive (often several Terabytes) and the "Mounted on" path is something like <code>/scratch</code>, <code>/data</code>, or <code>/local</code>. (Ignore drives mounted on <code>/home</code> or network shares).
			# '''Create your personal folder:''' You must have a dedicated place to store your files so you don't clutter the main directory. Once you have identified the scratch path, create a directory for your Andrew ID:
			<syntaxhighlight lang="bash">
			mkdir /scratch/your_andrew_id
			</syntaxhighlight>

	== 4. Special Note: Slurm-Managed Clusters ==		== 4. Special Note: Slurm-Managed Clusters ==

Ajshen: Created page with "When transitioning to remote server workflows, your first instinct might be to drag and drop files using the built-in file explorer in VS Code or other IDEs. While fine for sm..."

2026-06-27T06:17:04Z

Created page with "When transitioning to remote server workflows, your first instinct might be to drag and drop files using the built-in file explorer in VS Code or other IDEs. While fine for sm..."

New page

When transitioning to remote server workflows, your first instinct might be to drag and drop files using the built-in file explorer in VS Code or other IDEs. While fine for small scripts, GUI-based transfers quickly break down when moving gigabytes or terabytes of research data.

This guide covers the essential command-line tools for reliably moving data to and from the department servers.

== 1. The Limits of GUI-Based Transfers ==
Integrated Development Environments (IDEs) like VS Code Remote are incredible for writing code, but their built-in file uploaders are not built for big data. Relying on them for large datasets introduces several risks:

* '''No Checkpointing:''' If your Wi-Fi drops 90% of the way through a 50GB transfer, GUI tools usually force you to start over from the beginning.
* '''UI Freezing:''' Moving thousands of small files (like image datasets) can overwhelm the IDE's file watcher, causing your entire editor to freeze or crash.
* '''Memory Overhead:''' GUI tools often load chunks of the file into your local and remote RAM during transfer, which is incredibly inefficient and can trigger out-of-memory errors.

Command-line interface (CLI) tools bypass the visual overhead, offer robust error handling, and allow you to pause and resume massive transfers.

== 2. Essential CLI Transfer Tools ==
Below are the three core commands you will use for data management.

<blockquote>'''Prerequisite:''' For tools like <code>scp</code> and <code>rsync</code>, you must run these commands from your '''local machine's terminal''', not from inside the server.</blockquote>

=== <code>wget</code>: Downloading Directly to the Server ===
If your data is already hosted online (e.g., on GitHub, an open data portal, or a public AWS bucket), do not download it to your laptop first. Use <code>wget</code> to pull it directly onto the server at gigabit speeds.

'''Step-by-step Example:'''
# Connect to the server: <code>ssh your_stat_id@leviathan1.stat.cmu.edu</code>
# Navigate to your target folder: <code>cd /scratch/your_andrew_id/data</code>
# Download the file:
<syntaxhighlight lang="bash">
wget https://example-data-portal.com/massive_dataset.csv
</syntaxhighlight>

=== <code>scp</code> (Secure Copy): Best for Single Files ===
<code>scp</code> uses the same security protocol as SSH. It is the easiest way to push a specific file from your laptop to the server, or pull a results file back down.

'''Step-by-step Example:'''
# Open a terminal on your '''local computer'''.
# To push a file ''to'' the server:
<syntaxhighlight lang="bash">
scp ./local_model_weights.pt your_stat_id@leviathan1.stat.cmu.edu:/home/your_stat_id/models/
</syntaxhighlight>
# To pull a file ''from'' the server:
<syntaxhighlight lang="bash">
scp your_stat_id@leviathan1.stat.cmu.edu:/home/your_stat_id/results.csv ./Desktop/
</syntaxhighlight>

=== <code>rsync</code>: The Gold Standard for Big Data ===
If you are moving entire directories or datasets, <code>rsync</code> is mandatory. It checks what files already exist at the destination and only transfers the differences. If your internet dies, simply run the exact same <code>rsync</code> command again, and it will pick up right where it left off.

'''Step-by-step Example:'''
# Open a terminal on your '''local computer'''.
# Sync a local folder to the server. The flags <code>-avP</code> stand for Archive (keeps file structures intact), Verbose (shows progress), and Partial (allows resuming).
<syntaxhighlight lang="bash">
rsync -avP ./my_massive_dataset/ your_stat_id@leviathan1.stat.cmu.edu:/scratch/your_stat_id/dataset/
</syntaxhighlight>

See 🔗 https://www.geeksforgeeks.org/linux-unix/rsync-command-in-linux-with-examples/

== 3. Where to Put Your Data: Home vs. Scratch ==
Before you start moving terabytes of data, it is critical to understand ''where'' you are putting it. The department clusters have two distinct types of storage:

=== <code>/home</code> (Network Attached Storage) ===
Your home directory (e.g., <code>/home/your_andrew_id</code>) is hosted on a central Network Attached Storage (NAS) drive.
* '''The Benefit:''' Data stored here "follows" you. If you download a file to <code>/home</code> while logged into <code>leviathan1</code>, that exact same file will be waiting for you if you log into <code>leviathan2</code> or the <code>ghidorah</code> nodes.
* '''The Drawback:''' NAS drives have strict storage quotas and slower read/write speeds over the network. '''Do not put massive datasets here.'''

=== <code>/scratch</code> (Local Disk) ===
Every individual server node has its own dedicated physical hard drive, usually mounted at <code>/scratch</code>.
* '''The Benefit:''' Massive capacity and extremely fast disk I/O, meaning your code will read the data much faster. Always transfer your large datasets here.
* '''The Drawback:''' Scratch space is local to the machine. If you download a dataset to <code>/scratch/your_andrew_id/</code> on <code>leviathan1</code>, you '''cannot''' see or access it from <code>leviathan2</code>.

<blockquote>'''Best Practice:''' Keep your code, scripts, and small configuration files in <code>/home</code> so you can run them anywhere. Keep your large <code>.csv</code>, <code>.parquet</code>, or image files in <code>/scratch</code> on the specific node you are computing on.</blockquote>

== 4. Special Note: Slurm-Managed Clusters ==
If you are graduating from the Leviathans to massive shared clusters managed by '''Slurm''' (like the Wright GPU cluster or PSC), there is one strict rule regarding data transfer:

'''Never run heavy data transfers on the login nodes.'''

When you SSH into a Slurm cluster, you land on a shared "login node." If you run a massive <code>wget</code> or <code>rsync</code> command there, you will consume the network bandwidth for everyone else trying to log in, and system administrators may kill your process.

Instead, you must either:
# Start an interactive compute session (<code>salloc</code> or <code>srun</code>) and run your <code>wget</code>/<code>rsync</code> command from inside that active compute node.
# Check the cluster's documentation for a dedicated '''Data Transfer Node (DTN)''', which is a specific server address built exclusively to handle heavy <code>rsync</code> or Globus traffic.

Command Line Data Transfer - Revision history

Ajshen at 02:54, 29 June 2026

Ajshen: /* 3. Where to Put Your Data: Home vs. Scratch */

Ajshen: Created page with "When transitioning to remote server workflows, your first instinct might be to drag and drop files using the built-in file explorer in VS Code or other IDEs. While fine for sm..."