Linux and department servers

The department has two sets of servers, the Leviathans and the Hydras, that can help you run large analyses and gain a better understanding of common command line structure for more high scale computing. One can also view the department servers a bit more as a “sandbox” or “playground” as there are very few constraints and it’s okay to not run as polished code as you might on the PSC or Azure.

[Request: Windows users (and all users) - if you see comments that are outdated / less helpful than expected please contribute.]

What resources exist

1. The department provides 11 leviathan servers (named {leviathan1, ..., leviathan11}.stat.cmu.edu). Each Leviathan has 16 CPU cores and 128 GB of RAM. The Leviathan CPUs are much faster than the former Hydra CPUs.
2. The department provides 3 Ghidorah servers (named {ghidorah1, ..., ghidorah3}.stat.cmu.edu). Each Ghidorah has 64GB of RAM, 32 core processors and NVIDIA RTX 4090 GPU. Note that ghidorah3 is reserved for MSCF student capstone projects in the fall.
3. The Ghidorahs and Leviathans all share the same data file system. This allows you to upload a file to any of the servers and then have it on all of them.
4. OpenMPI is installed
   • This allows MPI based parallelization to be used
   • Note that the hydras ARE NOT network connected, so MPI can only be used within a single hydra. It cannot communicate across multiple hydras.
   • If you need to run something in parallel across multiple nodes, you should use PSC instead.

See What resources when for guides to help decide which resource you should be using.

Introduction to connecting to the server and useful Linux commands

The department servers all run Linux, and require you to use a command line to interact with it (as compared to Rstudio and other GUIs).

Getting Access: (ssh)

To get onto the server and type code you’ll need to access it through ssh.

• For Macs (and Linux), open up a terminal and type

  ssh andrew_id@leviathan1.stat.cmu.edu

  you will then have to type in your password (if you don’t know it or don’t know it or have never accessed the server before - email remarks@stat.cmu.edu).

• For Windows, there's a few options (with different pros and cons). Makeuseof provides an overview and introduction to how to get started on a few approaches, including PuTTY and Windows PowerShell. Again, make sure to email remarks@stat.cmu.edu to check that you have an account / or what your password is if you forgot.

If you’re accessing the servers a lot, you might also look into aliases and password storing for the ssh commands. - Mac/Linux: scotch.io - Windows: Powershell: microsoft, PuTTY: makeuseof

Passwordless ssh

To get on the server without typing in the password, you need to first generate a SSH key pair. This creates a private key that identifies your laptop, and a public key that the servers can use to recognize your private key.

• For Linux or Mac, open up a terminal and check whether you already have a key pair: run ls ~/.ssh and see if you have any files with names starting id_. If you do, you already have a key pair, and you can skip to the ssh-copy-id step below.
• Next, generate a key pair (both public and private key). There are different types of keys (RSA, ECDSA, and Ed25519). The -t option specifies the type of key pair.

ssh-keygen -t ed25519

• A prompt for what to name the key pair and what folder to place it in should appear. The username should be the username for your local computer and id_ed25519 is the key pair name. You can change it. The local key should be stored in the ~/.ssh folder on your local computer.

Generating public/private ed25519 key pair.
Enter file in which to save the key (/home/username/.ssh/id_ed25519):

• Hit Enter to accept the default name and location, or enter the folder and name manually.
• A prompt for a passphrase should appear. This is for securing the local key (on local computer) and is optional. Enter Passphrase (empty for no passphrase):

• Make sure there is a ~/.ssh folder on the server. If not, make a .ssh folder.
• Next, upload the public key to remote server. This command will upload the public key to all the hydras and ghidorahs since they share a file system.

ssh-copy-id andrew_id@leviathan1.stat.cmu.edu

• To ensure the permissions of the ~/.ssh directory are updated, log into the hydras and run:

chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

You should now be able to SSH into the server without being prompted for a password.

File sharing (scp +)

There is also command line code to copy files from your personal computer to the server. This is done with a SCP (secure copy) tool. There are also GUIs (graphical user interfaces) that allow you to “drag and drop” files from your computer to the server (and visa-versa). Currently, a lot of students use GUIs to deal with file sharing.

Command line scp

You can file share either by using scp, for example

1. Mac/Linux: This command is very similar to cp (the file copier).
   1. Copy from server

       scp username@from_host:path/to/file.txt /local/directory/

      scp -r username@from_host:path/to/folder/ local/directory/

   2. Copy to server  

       scp local/file.txt username@to_host:/remote/directory/

       scp -r local/directory/ username@to_host:/remote/directory/

   
   scp references: haydenjames.io

2. Windows: Again, this process is a bit more complicated.
   1. A PuTTY + Command Prompt provides a version that is very similar to Mac/Linux scp (reference: tanaza), and requires minor changes in the call  (pscp -scp instead of scp).
   2. A Powershell specific command is a bit more complicated / not similar to Mac/Linux scp (reference: nsoftware)

 GUI “drag-and-drop” file sharing

A lot of students use GUI interfaces to transfer files instead of scp command line. These tools allow you to “drag and drop” files from one location to another after the connection is set up.

Some examples include:

1. Mac:
   1. Cyberduck
2. Linux:
   1. OpenSSH
3. Windows: lots of options - thegeekstuff
   1. PuTTY
   2. WinSCP

Basic CLI (Linux command line): bash

Understanding the basic Linux command line is important to be able to navigate when you are actually on the server. Below are some based commands to move around the file system, manipulate files, and view storage structure.

command	translation
pwd	print the current working directory
cd _dir_	change directory
rm _file_	remove / delete file
mv _file_ _dir_	move file (can change name)
cp _file_ _dir_	copy file
ls _dir_	list files in directory

use man _command_ to get a listing / manual of the command.

Additional resources that capture more information on the command line and are still geared to statistical computing can be found here: UCB 2015 reproducible DS lecture notes.

TODO: add information about sed, chmod, and a link to bash files.

Using the servers respectfully

Because the servers are a shared department resource it is important to use them respectfully. There are many servers and multiple cores (CPUs) per server. To be respectful of other users:

1. Check the current usage of the server you're on before using it.
2. Do not use all the cores on a server, and do not use all the servers at the same time, so others can get work done.
3. If you have a long-running job, deprioritize it so others can run too.
4. Use /scratch for jobs that require intense disk I/O operations

Each of these is easy to check.

Checking current usage

To check which servers are currently being used and which are open, see the Grafana dashboard for the hydras and for the Leviathans. It lists them all and their current usage.

When you're logged in to one node, you can use the top command. top displays the current usage of the server’s resources with a list of jobs run. It also tells you the percentage of the cores that are being used and the amount of RAM (memory) that is available. See howtogeek resources on how to interpret it all (the documentation is very clean, so we encourage you to read up ther

Limiting the cores you use

We recommend that when using parallel processing, use fewer than the total number of cores on the server. Most parallel processing packages let you choose the number of cores to use; the hydras have 12, so pick something smaller.

Deprioritize when possible

nice/renice allows you to lower the priority of your jobs, so the operating system lets other processes run first. See howtogeek resources on the basics. nice sets the priority, and renice updates your priority. On the department servers, everyone starts at the same priority, and you can only deprioritize your jobs -- you can't make them higher priority than everyone else.

nice -20 Rscript some-big-script.R
renice -n 20 processid # get the process ID from top

Using /scratch

On department servers, your home directory (the default location you start in when you ssh) is not actually located on the physical server itself. Instead, every department server uses network attached storage (NAS), where everybody's home directly lives on a central storage server and the individual Leviathans and Ghidorahs connect to that storage server. This is the functionality that lets your files follow you no matter what node you log into.

While computing processes on one server do not affect processes on other servers, every server shares the same NAS. Therefore, if a user is reading and writing a ton of data to their home directory, it will slow down reading and writing for everyone, even if you aren't on the same server as this user. Even if your job does not read or write much data, disk I/O slowdowns will affect common operations, such as ssh, importing packages, and uploading/downloading files.

If you need to run a job that requires a ton of reading and writing data to disk, do not read/write using your home directory. Instead, use the /scratch folder located on every server. This folder points to an SSD that is specific to that server. Not only will read and writes to /scratch not affect users on other servers, but it will also be generally faster than using your home directory even if nobody else is online. If you plan to use /scratch, you should make a new folder inside of /scratch first and put your data inside that folder. For example:

cd /scratch
mkdir my_data # Default permission allows only you to modify contents
cd my_data
# Do stuff in my_data

It is also very important to remember that /scratch is specific to each Leviathan and Ghidorah and will not follow you between servers. Thus, you should not use /scratch for long-term storage or for important data. In addition, space on /scratch is limited, so be sure to clean up your data once you don't need it anymore.

Text editors

There are multiple options to view code on the server side. They include:

• vim
• … [to be added later]

Running jobs

We have included thoughts about running jobs in two places.

1. The first looks at tools that allow you to keep running jobs on the server when you close the connection to the server (turn off your laptop, etc.). These tools are call terminal multiplexers. 
2. The second set of tools focus on how to run your R analysis or Python analysis code on the server.

Server specs

Generally speaking, our servers have multiple CPU cores, but no GPU cores. For deep learning tasks that need GPUs, consider using the GPU cluster.

To examine the specs yourself, use

cat /proc/meminfo
cat /proc/cpuinfo

to see some information.