Linux and department servers

Scaling Checklist
Building Workflows on Remote Servers Parallelization: Making the most of your Machine Big Data Management
Back to Beginner's Guide

The department has two sets of servers, the Leviathans and the Hydras, that can help you run large analyses and gain a better understanding of common command line structure for more high scale computing.

Important: The Sandbox Philosophy
Think of the department servers as a “sandbox” or “playground.” There are very few constraints here, making it a great place to learn. It is perfectly fine to run code here that isn't quite polished enough for strict cluster environments like the PSC or Wright GPU Cluster.

[Request: All users - if you see comments that are outdated / less helpful than expected please contribute.]

What Resources Exist

1. The department provides 11 CPU servers (named leviathan1.stat.cmu.edu through leviathan11.stat.cmu.edu).
2. The department provides 3 GPU servers (named ghidorah1.stat.cmu.edu through ghidorah3.stat.cmu.edu).
3. The Ghidorahs and Leviathans all share the same network-attached data file system. This allows you to upload a file to any single server and instantly access it on all of them.

See What resources when for guides to help decide which resource you should be using.

Getting Access (SSH)

The department servers all run Linux and require you to interact with them remotely. To execute code, you’ll need to access them through ssh. Modern operating systems (Mac, Linux, and Windows 10/11) have this tool built-in natively.

• Mac / Linux: Open up your Terminal.
• Windows: Open up Windows PowerShell or Windows Terminal.

In your terminal, type:

ssh andrew_id@leviathan1.stat.cmu.edu

You will be prompted to type in your password. (Note: as you type, no characters will appear on the screen for security reasons. Just type it and hit Enter). If you don’t know your password or have never accessed the server before, email remarks@stat.cmu.edu.

Passwordless SSH (Highly Recommended)

To connect to the server without typing in your password every time, you need to generate an SSH key pair.

1. Check whether you already have a key pair by looking in your .ssh folder (run ls ~/.ssh on Mac/Linux, or dir ~\.ssh on Windows). If you see id_ed25519 or id_rsa, you can skip to step 3.
2. To generate a new key pair, run:

   ssh-keygen -t ed25519

   Hit Enter to accept the default file location. You can enter a passphrase for extra security or leave it empty for true passwordless login.
3. Upload the public key to the remote server (you only need to do this once since the servers share a file system).
   • For Mac/Linux:
     

     ssh-copy-id andrew_id@leviathan1.stat.cmu.edu

   • For Windows: Windows does not natively include ssh-copy-id. Use this PowerShell one-liner instead:
     

     Get-Content "$env:USERPROFILE\.ssh\id_ed25519.pub" | ssh andrew_id@leviathan1.stat.cmu.edu "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys"

4. Finally, log into the server normally and run the following to enforce strict permissions:

5. chmod 700 ~/.ssh
   chmod 600 ~/.ssh/authorized_keys

File Sharing & Management

There are two primary ways to move files between your personal computer and the server.

Command Line (scp)

The scp (secure copy) command works identically across Mac, Linux, and Windows PowerShell.

Strengths	Weaknesses
Fast and Native: Requires no additional software installations. Scriptable: Can be built into bash scripts to automate backing up data.	Syntax Heavy: Easy to forget the exact command structure or path names. No Visual Feedback: Doesn't offer a traditional file-explorer view of the remote server.

From server to your computer:

scp andrew_id@leviathan1.stat.cmu.edu:/path/to/remote/file.txt /local/directory/

• (To copy an entire folder, add the -r flag right after scp).*

From your computer to the server:

scp /local/file.txt andrew_id@leviathan1.stat.cmu.edu:/remote/directory/

GUI "Drag-and-Drop" Tools

Many students prefer graphical interfaces to transfer files.

• Visual Studio Code (Recommended): If you use the VS Code Remote-SSH extension, you can simply drag and drop files from your laptop directly into the VS Code file explorer pane.
• Mac: Cyberduck remains a solid dedicated tool.
• Windows: WinSCP is a popular dedicated GUI (legacy SSH clients like PuTTY are no longer necessary).

Basic Linux Command Line (Bash)

Understanding the basic Linux command line is crucial for navigating the servers. Use man _command_ to get the manual for any command.

Command	Translation	Example Usage
pwd	Print working directory	Shows exactly where you are in the file system.
ls _dir_	List files	ls -l shows file permissions and sizes.
cd _dir_	Change directory	cd .. moves you up one folder.
rm _file_	Remove / delete file	rm -r _folder_ deletes an entire directory. (Irreversible!)
mv _file_ _dir_	Move / rename file	mv data.csv ./cleaned_data.csv
cp _file_ _dir_	Copy file	cp script.R backup_script.R
chmod _perms_ _file_	Change permissions	chmod +x script.sh makes a script executable.
sed	Stream editor	sed -i 's/old/new/g' file.txt replaces text inside a file automatically.

For a deeper dive into shell scripting and Linux environments, we recommend the UCB reproducible DS lecture notes or the Linux Documentation Project Bash Guide.

Text Editors and IDEs for writing code on the servers

Historically, editing code on a server required terminal-based text editors. While still useful, modern tools allow you to use a full graphical IDE on your laptop that computes natively on the server.

Visual Studio Code (Highly Recommended)

Visual Studio Code is the industry standard for remote development.

1. Install VS Code on your personal machine.
2. Go to the Extensions tab and install Remote - SSH.
3. Click the green remote indicator in the bottom-left corner, select "Connect to Host...", and type andrew_id@leviathan1.stat.cmu.edu.
4. Once connected, you can edit scripts, run interactive terminal sessions, and drag-and-drop files exactly as if they were on your laptop.

Terminal Editors

If you only need to make a quick tweak to a file while already logged into the terminal:

nano	vim
A very simple, beginner-friendly text editor. Commands are listed at the bottom of the screen. Type nano _filename_ to open. Use Ctrl+O to save (Write Out). Use Ctrl+X to exit.	A highly powerful, ubiquitous editor with a steep learning curve. Type vim _filename_ to open. Press i to enter Insert Mode (to type). Press Esc then type :wq to save and quit.

Using the Servers Respectfully

Because the servers are a shared department resource, it is critical to use them considerately.

The Golden Rules of Shared Servers:
1. Check current usage before launching massive scripts.
2. Do not use all the cores on a single server.
3. Do not use all the servers at the exact same time.
4. Deprioritize long jobs so active users aren't frozen out.

1. Checking Current Usage

Check the Grafana dashboard for the Hydras and the Grafana dashboard for the Leviathans to see active usage across all nodes.

Once logged into a specific node, run the top command. This displays the percentage of cores being used, available RAM, and a list of running jobs. Press q to exit top. (See this HowToGeek guide for interpreting the output).

2. Limiting Your Cores

When using parallel processing packages, explicitly set the number of cores to less than the server's maximum. For example, the hydras have 12 cores—hardcode your script to use 8 or fewer so the OS has room to breathe.

3. Deprioritize When Possible (nice/renice)

nice allows you to lower the priority of your jobs, telling the OS to let other processes run first. Everyone starts at the same priority level, and you can only deprioritize your own jobs.

# Start a new job with lowered priority
nice -20 Rscript some-big-script.R

# Lower the priority of a job that is already running
renice -n 20 processid # get the process ID from the 'top' command

4. Using /scratch for Heavy Data

CRITICAL: Do not run heavy Read/Write jobs from your Home directory!
Your home directory lives on a central network-attached storage (NAS) drive. If you read/write massive datasets directly to ~, you will bottleneck the network and slow down operations (even basic SSH access) for every single user across all servers.

Instead, use the /scratch folder. This folder points to a local, lightning-fast SSD specific to the server you are currently logged into.

How to use /scratch:

cd /scratch
mkdir andrew_id_data # Create a folder for yourself
cd andrew_id_data
# Move your data here and run your heavy I/O scripts!

Important Caveats for /scratch:

• It is specific to the node you are on (leviathan1's scratch is different from leviathan2's scratch).
• It is strictly for temporary runtime storage. Do not use it for long-term storage.
• Space is limited. Delete your files when your script finishes running.

Running Jobs & Multiplexers

If you start a script over standard SSH and close your laptop, the connection breaks and your job is instantly killed.

To run jobs that persist after you disconnect, you must use a terminal multiplexer like tmux or screen. These tools create a virtual session on the server that keeps running in the background.

• Start a new session: tmux new -s my_analysis
• Detach (leave it running in background): Press Ctrl+b, release, then press d
• Reattach later: tmux attach -t my_analysis

For language-specific guides on running batch analyses, see our dedicated pages for R analysis and Python analysis. For more advanced multiplexer features, see terminal multiplexers.

Server Specs

Our servers have multiple CPU cores, but only the ghidorah servers have GPU cores. For deep learning tasks that require GPUs, consider using the GPU Cluster.

To check the hardware specifications of the server you are currently logged into, run:

cat /proc/cpuinfo # View processor details and core counts
cat /proc/meminfo # View RAM and memory allocation