Linux and department servers

From Stat DS Computing Wiki


The department has two sets of servers, the Leviathans and the Hydras, that can help you run large analyses and gain a better understanding of common command line structure for more high scale computing. One can also view the department servers a bit more as a “sandbox” or “playground” as there are very few constraints and it’s okay to not run as polished code as you might on the PSC or Wright GPU Cluster.

[Request: All users - if you see comments that are outdated / less helpful than expected please contribute.]

What resources exist

  1. The department provides 11 leviathan servers (named {leviathan1, ..., leviathan11}.stat.cmu.edu).
  2. The department provides 3 Ghidorah GPU servers (named {ghidorah1, ..., ghidorah3}.stat.cmu.edu).
  3. The Ghidorahs and Leviathans all share the same data file system. This allows you to upload a file to any of the servers and then have it on all of them.

See What resources when for guides to help decide which resource you should be using.

Introduction to connecting to the server and useful Linux commands

The department servers all run Linux and require you to interact with them remotely. While you can use a traditional command line, modern IDEs like VS Code also allow for highly integrated graphical interaction.

Getting Access: (ssh)

To get onto the server and execute code, you’ll need to access it through ssh. Modern operating systems (including Mac, Linux, and Windows 10/11) have this tool built-in natively.

  • Mac / Linux: Open up a Terminal.
  • Windows: Open up Windows PowerShell or Windows Terminal.

In your terminal, type: 

ssh andrew_id@leviathan1.stat.cmu.edu

You will then be prompted to type in your password. (Note: as you type, no characters will appear on the screen for security reasons. Just type it and hit Enter). If you don’t know your password or have never accessed the server before, email remarks@stat.cmu.edu.

If you’re accessing the servers frequently, we highly recommend setting up passwordless SSH (see below).

Passwordless SSH

To connect to the server without typing in your password every time, you need to generate an SSH key pair. This creates a private key that stays securely on your laptop, and a public key that the servers use to recognize you.

  • Open up your terminal (Mac/Linux) or PowerShell (Windows).
  • Check whether you already have a key pair by looking in your .ssh folder (run ls ~/.ssh on Mac/Linux, or dir ~\.ssh on Windows). If you have files like id_ed25519 or id_rsa, you already have a key pair and can skip to the upload step.
  • To generate a new key pair, we recommend the Ed25519 algorithm. Run:
ssh-keygen -t ed25519
  • Hit Enter to accept the default file location. When prompted for a passphrase, you can enter one for extra security or leave it empty for true passwordless login.
  • Next, upload the public key to the remote server. Because the hydras and ghidorahs share a file system, you only need to do this once.
    • For Mac/Linux: Use the built-in ssh-copy-id command:
    • ssh-copy-id andrew_id@leviathan1.stat.cmu.edu
    • For Windows: Windows OpenSSH does not natively include the ssh-copy-id command. Instead, use this PowerShell one-liner to securely copy your key over:
    • Get-Content "$env:USERPROFILE\.ssh\id_ed25519.pub" | ssh andrew_id@leviathan1.stat.cmu.edu "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys"
  • To ensure the permissions of the ~/.ssh directory on the server are strictly enforced, log into the server normally and run:
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

You should now be able to SSH into the server without being prompted for a password.

File sharing (scp + GUIs)

There are two primary ways to move files between your personal computer and the server: using the command line (scp) or using a graphical interface (like VS Code or Cyberduck).

Command line scp

Because Windows now includes OpenSSH natively, the scp (secure copy) command works exactly the same across Mac, Linux, and Windows PowerShell.

  1. Copy from server to your computer 
    scp andrew_id@leviathan1.stat.cmu.edu:/path/to/remote/file.txt /local/directory/
     To copy an entire folder, add the -r (recursive) flag:

    scp -r andrew_id@leviathan1.stat.cmu.edu:/path/to/remote/folder/ /local/directory/
  2. Copy from your computer to the server 
    scp /local/file.txt andrew_id@leviathan1.stat.cmu.edu:/remote/directory/

GUI “drag-and-drop” file sharing

Many students prefer graphical user interfaces (GUIs) to transfer files instead of using the command line. These tools allow you to visually drag and drop files from your computer to the server (and vice-versa).

  • Visual Studio Code (Recommended for all OS): If you use VS Code with the Remote-SSH extension (see "Text Editors" below), you can simply drag and drop files from your desktop directly into the VS Code file explorer pane. This is the easiest and most integrated method.
  • Mac: Cyberduck remains a solid dedicated tool.
  • Windows: WinSCP is a popular dedicated GUI, though legacy SSH clients like PuTTY are no longer necessary for modern Windows file transfers.

Basic CLI (Linux command line): bash

Understanding the basic Linux command line is important to navigate when you are on the server. Below are some basic commands to move around the file system and manipulate files.

command translation

pwd

print the current working directory

cd _dir_

change directory

rm _file_

remove / delete file

mv _file_ _dir_

move file (can change name)

cp _file_ _dir_

copy file

ls _dir_

list files in directory

Use man _command_ to get a listing / manual of the command.

Additional resources geared toward statistical computing can be found here: UCB 2015 reproducible DS lecture notes.

TODO: add information about sed, chmod, and a link to bash files.

Text editors and IDEs

Historically, editing code on a server required using terminal-based text editors. While still useful for quick changes, modern tools allow you to use a full graphical IDE on your laptop that computes natively on the server.

Visual Studio Code (Highly Recommended)

Visual Studio Code (VS Code) is the industry standard for remote development. By using the Microsoft Remote - SSH extension, you can connect VS Code directly to the department servers.

  1. Install VS Code on your personal machine.
  2. Go to the Extensions tab and install Remote - SSH.
  3. Click the remote indicator (the green arrows) in the bottom-left corner of VS Code, select "Connect to Host...", and type andrew_id@leviathan1.stat.cmu.edu.
  4. Once connected, you can open server folders, edit scripts with full syntax highlighting, run interactive terminal sessions, and drag-and-drop files between your laptop and the server all from within the VS Code window.

Terminal Editors

If you only need to make a quick tweak to a file while already logged into the terminal, you can use built-in editors:

  • nano _filename_: A very simple, beginner-friendly text editor. Use Ctrl+O to save and Ctrl+X to exit.
  • vim _filename_: A highly powerful but steep-learning-curve text editor.

Using the servers respectfully

Because the servers are a shared department resource it is important to use them respectfully. There are many servers and multiple cores (CPUs) per server. To be respectful of other users:

  1. Check the current usage of the server you're on before using it.
  2. Do not use all the cores on a server, and do not use all the servers at the same time, so others can get work done.
  3. If you have a long-running job, deprioritize it so others can run too.
  4. Use /scratch for jobs that require intense disk I/O operations

Each of these is easy to check.

Checking current usage

To check which servers are currently being used and which are open, see the Grafana dashboard for the hydras and for the Leviathans. It lists them all and their current usage.

When you're logged in to one node, you can use the top command. top displays the current usage of the server’s resources with a list of jobs run. It also tells you the percentage of the cores that are being used and the amount of RAM (memory) that is available. See howtogeek resources on how to interpret it all.

Limiting the cores you use

We recommend that when using parallel processing, use fewer than the total number of cores on the server. Most parallel processing packages let you choose the number of cores to use; the hydras have 12, so pick something smaller.

Deprioritize when possible

nice/renice allows you to lower the priority of your jobs, so the operating system lets other processes run first. See howtogeek resources on the basics. nice sets the priority, and renice updates your priority. On the department servers, everyone starts at the same priority, and you can only deprioritize your jobs -- you can't make them higher priority than everyone else.

nice -20 Rscript some-big-script.R
renice -n 20 processid # get the process ID from top

Using /scratch

On department servers, your home directory (the default location you start in when you ssh) is not actually located on the physical server itself. Instead, every department server uses network attached storage (NAS), where everybody's home directly lives on a central storage server and the individual Leviathans and Ghidorahs connect to that storage server. This is the functionality that lets your files follow you no matter what node you log into.

While computing processes on one server do not affect processes on other servers, every server shares the same NAS. Therefore, if a user is reading and writing a ton of data to their home directory, it will slow down reading and writing for everyone, even if you aren't on the same server as this user. Even if your job does not read or write much data, disk I/O slowdowns will affect common operations, such as ssh, importing packages, and uploading/downloading files.

If you need to run a job that requires a ton of reading and writing data to disk, do not read/write using your home directory. Instead, use the /scratch folder located on every server. This folder points to an SSD that is specific to that server. Not only will read and writes to /scratch not affect users on other servers, but it will also be generally faster than using your home directory even if nobody else is online. If you plan to use /scratch, you should make a new folder inside of /scratch first and put your data inside that folder. For example:

cd /scratch
mkdir my_data # Default permission allows only you to modify contents
cd my_data
# Do stuff in my_data

It is also very important to remember that /scratch is specific to each Leviathan and Ghidorah and will not follow you between servers. Thus, you should not use /scratch for long-term storage or for important data. In addition, space on /scratch is limited, so be sure to clean up your data once you don't need it anymore.

Running jobs

We have included thoughts about running jobs in two places.

  1. The first looks at tools that allow you to keep running jobs on the server when you close the connection to the server (turn off your laptop, etc.). These tools are call terminal multiplexers. 
  2. The second set of tools focus on how to run your R analysis or Python analysis code on the server.

Server specs

Generally speaking, our servers have multiple CPU cores, but no GPU cores. For deep learning tasks that need GPUs, consider using the GPU cluster.

To examine the specs yourself, use 

cat /proc/meminfo
cat /proc/cpuinfo

to see some information.

Retrieved from "https://info.stat.cmu.edu/index.php?title=Linux_and_department_servers&oldid=920"