Wright GPU Cluster

The department has access to a powerful GPU-based cluster named Wright which is hosted here at CMU. This is a 3-node and growing cluster that is a shared resource for Statistics & Data Science, Philosophy, and other departments in Dietrich College. Each department has priority access to its own compute node, but will be able to share the resources from other departments when not in use.

ACCESS REQUEST FORM

This resource is free to use to everyone in the Departments of Statistics & Data Science and Philosophy.  Users outside these departments must be sponsored by someone in the eligible departments, as part of a project or collaboration.

As the cluster is brand new and we’re still building our expertise (and documentation) and working out quirks, we do need to do a little gatekeeping.  If you anticipate that this is a necessary resource for you, please complete the following form and our team will be in touch to discuss your request and onboard you.

Cluster setup

The cluster currently consists of 4 servers:

• The head node: You log in to this to write code and run jobs. It has 128 GB of RAM, 16 cores, and no GPU.
• The compute nodes: Jobs run on these. Each has 256 CPU cores, 8 NVIDIA L40 GPU cards with 48 GB of video RAM each, approximately 1 TB of RAM, and a total of approximately 3 TB of local data/scratch space.
• The file server: Holds 50TB of storage. The head node and compute nodes connect via fiberoptic networking. Your home directory is hosted on the file server, so the same files are accessible on the head node and all compute nodes.

The cluster is currently partitioned. One compute node is reserved for Statistics & Data Science and one is reserved for Philosophy.

Connecting and setting up your environment

SSH

From your terminal, type ssh <YourStatID>@wright.hss.cmu.edu . You can also avoid remembering this by adding the following snippet to your local ~/.ssh/config file:

Host wright
    Hostname wright.hss.cmu.edu
    User YourStatID

You can then connect by simply typing ssh wright.

This connects you to the head node. Here you can upload your data files, set up your scripts, and test things.

Do not run resource-intensive tasks directly on the head node. It doesn't have any GPUs, and it has fewer resources than the compute nodes. You can do quick tests to make sure your scripts are set up correctly, but submit all other jobs through Slurm.

Uploading files

To upload scripts or data files, you can use scp from your local computer (Mac or Linux); on Windows, you can get the Tectia SSH graphical client free through CMU.

You can also connect through the Remote SSH feature of VS Code if you like. This is useful to 

• navigate the file system from a proper GUI and 
• modify code/files/folder in place if you are not familiar with the command line

Files uploaded to your home directory will be accessible on the head node and all compute nodes, so they can be freely used in compute jobs.

Using Conda

The head node has Anaconda installed for managing Python packages and system libraries; see Python virtual environments for information on how to use it (but note it's already installed, so you don't need to install it from scratch!).

However, the compute nodes do not have conda, so your slurm jobs can't use conda activate or other familiar commands. Instead, set up your environment on the head node, activate it, install the packages you need, and so on. Then run which python to get the path to the Python executable in your environment. For instance:

$ which python
/home/export/areinhar/.conda/envs/llm/bin/python

This is the path you'll use in your scripts (see below).

Running jobs

Slurm

The GPU cluster manages jobs via Slurm. Slurm is a job management system that deals with job prioritization and the allocation of resources. On high-performance computers (e.g., PSC and our GPU cluster) Slurm helps allocate resources so multiple jobs can be run quickly. Users run programs by submitting scripts to the Slurm job scheduler, and the scheduler runs them when resources are available.

To check what jobs are running, use squeue:

$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
              1326      phil     bash shaoanxi  R   22:41:00      1 n02
              1327      phil     bash shaoanxi  R   22:39:42      1 n02
              1486    statds run_coun  soheuny  R    1:12:53      1 n01
              1488    statds run_coun  soheuny  R    1:06:00      1 n01
              1489    statds run_coun  soheuny  R    1:00:53      1 n01
              1490    statds run_coun  soheuny  R    1:00:07      1 n01
              1491    statds run_coun  soheuny  R      24:02      1 n01
              1492    statds run_coun  soheuny  R       6:15      1 n01
              1493    statds run_coun  soheuny  R       3:16      1 n01
              1494    statds run_coun  soheuny  R       0:52      1 n01

Here we see there are eight jobs running on node 1 and 2 jobs running on node 2.

To check what resources are currently in use, use sinfo:

$ sinfo -O "NodeList:10,CPUsState:16,Memory:9,FreeMem:10,Gres:20,GresUsed:30"
NODELIST  CPUS(A/I/O/T)   MEMORY   FREE_MEM  GRES                GRES_USED                     
n01       64/192/0/256    1000000  221943    gpu:nvidia-l40:8    gpu:nvidia-l40:8(IDX:0-7)     
n02       32/224/0/256    1000000  225575    gpu:nvidia-l40:8    gpu:nvidia-l40:0(IDX:N/A)

This shows that node 1 (Statistics & Data Science) has 64 CPUs allocated and 192 idle, out of 256 total. There is 1TB of RAM, of which 222 GB is free. There are 8 NVIDIA L40 GPus, all 8 of which are in use. Hence if we submit a job to the statds partition requesting any GPUs, it will have to wait until some become free. Meanwhile, node 2 (Philosophy) has no GPUs in use.

Job queuing and scheduling

The cluster is divided into two partitions: statds (Statistics & Data Science) and phil (Philosophy). Jobs are submitted to a partition. Users should submit jobs to their department's partition -- jobs submitted to the wrong partition may be killed.

Slurm runs jobs based on the resources they request and what is available. If resources aren't available for your job, slurm will put it in the queue and wait until other jobs finish. However, it's not simply a first-in first-out queue -- if it were, large jobs requiring lots of resources would never run as long as smaller jobs were using up the resources. Instead, a complex scheduling and priority system picks which jobs to run when.

Examples:

• Someone submits Job A that requests 6 GPUs. Later, you submit Job B that requests 4 GPUs on the same partition. The compute node only has 8 GPUs, so Job B will be queued until Job A completes.
• Someone submits Job A that requests 6 GPUs. Later, you submit Job B that requests 2 GPUs. Job B can run immediately.
• You submit a job that requires all 8 GPUs on a node, but several are currently in use. Your job is queued. Based on the expected completion time of the current jobs, slurm will schedule your job to run when they're done. Jobs submitted after yours may or may not run first, depending on their priority and how long they're expected to run.

You can view the currently running or queued jobs by running squeue on the head node. For example:

$ squeue
     JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
      1116      phil     bash shaoanxi  R 1-03:06:57      1 n02
      1117      phil     bash shaoanxi  R 1-03:06:48      1 n02
      1224    statds    j_job  jacobg2  R       0:01      1 n01

Here we see that the philosophy partition has two jobs running, while the Statistics & Data Science partition has one. If a job is delayed because resources are not available, the NODELIST(REASON) column will display (Resources).

Batch Mode

This is the standard mode to launch jobs via a script. To launch a batch job from a script you will need two files:

• the actual script that implements things in your favorite programming language
• a bash script that requests the resources you need from Slurm and gives the commands to launch your job.

A typical bash script might look like this:

#!/bin/bash
# specify partition to run in (statds or phil)
#SBATCH --partition statds
#SBATCH --ntasks 1

# number of CPUs you need:
#SBATCH --cpus-per-task 1

# amount of memory per CPU
# (total allocated is CPUs * memory)
#SBATCH --mem-per-cpu 16G

# number of GPUs you need
#SBATCH --gres=gpu:1

# maximum time limit for task (up to 48 hours max, 48:00:00)
#SBATCH --time=01:00:00

# email you when job is finished:
#SBATCH --mail-user=YourAndrewID@andrew.cmu.edu
#SBATCH --mail-type=END

## Choose one:

# to run a Python script in a Conda environment called ENV
srun /home/export/YOURANDREWID/.conda/envs/ENV/bin/python YourScript.py

# to run an R script:
srun Rscript YourScript.R

Adjust the number of CPUs, number of GPUs, memory usage, and time limit based on the needs of your script. Requesting more than you need may result in your job having to wait for resources to free up; requesting a long time you don't need will block slurm from scheduling other jobs for when yours will be done. (And if you request more resources than exist on the compute nodes, your job will be queued forever while it waits for resources to appear.)

Note: At the moment, Slurm jobs are limited to a maximum duration of 48 hours. Jobs running longer will be killed.

To run the job, SSH into the head node and navigate to the directory with your scripts. Then run:

sbatch example.sh

where example.sh is the name of your bash script.

You will see a message like "Submitted batch job 1209" when your job is submitted. Any printed output from your script will automatically be redirected to a file based on the job number -- in this case, slurm-1209.out. You can use tail -f slurm-1209.out to view the output live and monitor your script's progress.

Interactive Mode

You can also run interactive sessions that allow you to, say, run an R or Python console and type commands and immediately view the output. From the head node, run a command like:

# remember to edit to correct partition:
srun --partition statds --ntasks 1 \
    --cpus-per-task 16 --mem-per-cpu 16G \
    --gres=gpu:2 --time=00:05:00 --pty \
    bash

This runs a job that opens a bash shell on the compute node, and connects you directly to this job. You can now run bash commands, open Python, or run R scripts. Note your interactive session has a time limit and will be killed when it elapses, so set the time limit carefully and choose the resource amounts in the command according to your needs.

Keep the session time limit short: If you set a long limit (like the maximum 48 hours), open the session in tmux, and forget about it, you will block others from using those resources for 48 hours.

Interactive via VS Code

VS Code supports SSH and tunneling, which allows it to connect to a server and run commands (including Jupyter notebooks). To use this, however, we need to connect to the head node, run an interactive Slurm session, and then connect VS Code to that Slurm session so it can use it. This takes a few extra steps.

Setup

Before you use this for the first time, do the following steps:

• SSH into Wright and run the following to install and unpack the VSCode CLI: 

curl -Lk 'https://code.visualstudio.com/sha/download?build=stable&os=cli-alpine-x64' --output vscode_cli.tar.gz
tar -xf vscode_cli.tar.gz

• Add the code executable to your PATH by adding the following to your .bashrc file: export PATH="~:$PATH"
• In VSCode, install the Remote - Tunnels extension, then use the VSCode SSH feature to connect to Wright and make sure to install all the local VS Code extensions on the remote machine as well (e.g., Python, Jupyter, ...)

Running a session

• In the Wright headnode, launch a Slurm interactive session with srun, following the instructions above.

• Inside the interactive session, run code tunnel and follow the instructions. The first time the system will ask you to name the tunnel and authorize the process via an authenticator, I suggest using GitHub since it usually works quite smoothly. Subsequent tunnels might still ask you to authorize their creation by typing an alphanumerical sequence into a link that appears in the terminal. Once you’ve done that, the tunnel is successfully created.
• From VSCode, select Connect to Tunnel, then choose the authentication method you previously used and you will see the tunnel appear in the dropdown. Select it and you will land directly into the interactive session we just created
• You will now be able to run code and, e.g., Jupyter Notebooks using the compute node allocated by the Slurm interactive session. VS Code will be disconnected when the Slurm session ends, such as when it reaches its time limit.

Note that if you need to use a conda environment, you'll have to configure VS Code to use the Python executable you found above, since conda is not installed on the compute nodes.