Wright GPU Cluster

The department has access to a powerful GPU-based cluster named Wright which is hosted here at CMU. This is a 4-node and growing cluster that is a shared resource for Statistics & Data Science, Philosophy, and other departments in Dietrich College. Each department has priority access to its own compute node, but will be able to share the resources from other departments when not in use.

Access requests

This resource is free to use to users in the Departments of Statistics & Data Science, Philosophy, and CMIST. These departments set policies determining which of their users are eligible. Users outside these departments must be sponsored by someone in the eligible departments, as part of a project or collaboration.

To request access, complete the access request form and our team will be in touch to discuss your request and onboard you.

Cluster setup

The cluster currently consists of 4 servers:

• The head node: You log in to this to write code and run jobs. It has 128 GB of RAM, 16 cores, and no GPU.
• The compute nodes: Jobs run on these. Each has 256 CPU cores, 8 NVIDIA L40 GPU cards with 48 GB of video RAM each, approximately 1 TB of RAM, and a total of approximately 3 TB of local data/scratch space.
• The file server: Holds 50TB of storage. The head node and compute nodes connect via fiberoptic networking. Your home directory is hosted on the file server, so the same files are accessible on the head node and all compute nodes.

The cluster is currently partitioned. One compute node is reserved for each department involved.

The file server has quotas enabled so that each department gets 1/3rd of the available storage. If a department user tries to add files beyond the quota, the writes will be rejected, and nobody in that department will be able to add files until space is freed. To check the quotas and your own usage, view the quota report.

Connecting and setting up your environment

SSH

From your terminal, type ssh <YourStatID>@wright.hss.cmu.edu . You can also avoid remembering this by adding the following snippet to your local ~/.ssh/config file:

Host wright
    Hostname wright.hss.cmu.edu
    User YourStatID

You can then connect by simply typing ssh wright.

This connects you to the head node. Here you can upload your data files, set up your scripts, and test things.

Do not run resource-intensive tasks directly on the head node. It doesn't have any GPUs, and it has fewer resources than the compute nodes. You can do quick tests to make sure your scripts are set up correctly, but submit all other jobs through Slurm.

Uploading files

To upload scripts or data files, you can use scp from your local computer (Mac or Linux); on Windows, you can get the Tectia SSH graphical client free through CMU.

You can also connect through the Remote SSH feature of VS Code if you like. This is useful to 

• navigate the file system from a proper GUI and 
• modify code/files/folder in place if you are not familiar with the command line

Files uploaded to your home directory will be accessible on the head node and all compute nodes, so they can be freely used in compute jobs.

Using Conda

The head node has Anaconda installed for managing Python packages and system libraries; see Python virtual environments for information on how to use it (but note it's already installed, so you don't need to install it from scratch!).

However, the compute nodes do not have conda, so your slurm jobs can't use conda activate or other familiar commands. Instead, set up your environment on the head node, activate it, install the packages you need, and so on. Then run which python to get the path to the Python executable in your environment. For instance:

$ which python
/home/export/areinhar/.conda/envs/llm/bin/python

This is the path you'll use in your scripts (see below).

Open OnDemand

Open OnDemand is a graphical interface where you can run jobs, interactive applications, and manage files. More information can be found here.

Job scheduling and prioritization

The GPU cluster manages jobs via Slurm. Slurm is a job management system that deals with job prioritization and the allocation of resources. On high-performance computers (e.g., PSC and our GPU cluster) Slurm helps allocate resources so multiple jobs can be run quickly. Users run programs by submitting scripts to the Slurm job scheduler, and the scheduler runs them when resources are available.

Queue status

To check what jobs are running, use squeue:

$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
              1326      phil     bash shaoanxi  R   22:41:00      1 n02
              1327      phil     bash shaoanxi  R   22:39:42      1 n02
              1486    statds run_coun  soheuny  R    1:12:53      1 n01
              1488    statds run_coun  soheuny  R    1:06:00      1 n01
              1489    statds run_coun  soheuny  R    1:00:53      1 n01
              1490    statds run_coun  soheuny  R    1:00:07      1 n01
              1491    statds run_coun  soheuny  R      24:02      1 n01
              1492    statds run_coun  soheuny  R       6:15      1 n01
              1493    statds run_coun  soheuny  R       3:16      1 n01
              1494    statds run_coun  soheuny  R       0:52      1 n01

Here we see there are eight jobs running on node 1 and 2 jobs running on node 2.

To check what resources are currently in use, use sinfo:

$ sinfo -O "NodeList:10,CPUsState:16,Memory:9,FreeMem:10,Gres:20,GresUsed:30"
NODELIST  CPUS(A/I/O/T)   MEMORY   FREE_MEM  GRES                GRES_USED                     
n01       64/192/0/256    1000000  221943    gpu:nvidia-l40:8    gpu:nvidia-l40:8(IDX:0-7)     
n02       32/224/0/256    1000000  225575    gpu:nvidia-l40:8    gpu:nvidia-l40:0(IDX:N/A)

This shows that node 1 (Statistics & Data Science) has 64 CPUs allocated and 192 idle, out of 256 total. There is 1TB of RAM, of which 222 GB is free. There are 8 NVIDIA L40 GPus, all 8 of which are in use. Hence if we submit a job to the statds partition requesting any GPUs, it will have to wait until some become free. Meanwhile, node 2 (Philosophy) has no GPUs in use.

Partitions and condos

The cluster is divided into three partitions:

• statds (Statistics & Data Science)
• phil (Philosophy)
• cmist (CMIST)

Jobs are submitted to a partition. Users must submit jobs to their department's partition -- jobs submitted to the wrong partition will be rejected.

There are several other special partitions:

• all requests any available node from any partition. Thus a Philosophy user can run a job on the CMIST node, or vice versa.
• statds_condo, phil_condo, and cmist_condo are "condo" partitions. Any user can submit to any of these partitions.

These special partitions are like holiday condos: any user can access them, but with a lower priority than the owner (the host department). Jobs running on them are also preemptible. For example, if a Philosophy user submits to statds_condo, their job will run only when Statistics users are not using the resources. If a Statistics user submits a large job after yours, your job might be terminated, and will automatically run again when resources become available. The all partition is similar, but for whichever node happens to be free.

Job prioritization

Slurm runs jobs based on the resources they request and what is available. If resources aren't available for your job, slurm will put it in the queue and wait until other jobs finish. However, it's not simply a first-in first-out queue -- Slurm attempts to arrange the jobs so every job can run, even if that means running them out of order. Slurm also adjusts each user's priority based on how many jobs they submit, so that one user can't hog the cluster by submitting dozens of jobs; if one user has many jobs waiting, theirs will be reduced in priority so other users can run jobs as well.

Examples:

• Someone submits Job A that requests 6 GPUs. Later, you submit Job B that requests 4 GPUs on the same partition. The compute node only has 8 GPUs, so Job B will be queued until Job A completes.
• Someone submits Job A that requests 6 GPUs. Later, you submit Job B that requests 2 GPUs. Job B can run immediately.
• You submit a job that requires all 8 GPUs on a node, but 4 are currently in use. Your job is put in the queue and scheduled to run when Slurm knows the current jobs will finish (because of their maximum time limit). Meanwhile, someone submits a job using 2 GPUs and a short time limit. Slurm lets it run immediately, because it knows it will be done before your job is scheduled to run. Other jobs may be put off until after yours runs, so yours can get the 8 GPUs it needs.
• You submit 50 jobs that each require 1 GPU. The node has only 8 GPUs, but several are in use. Several of your jobs run, but the others are queued for later. Other users can submit jobs while yours are in the queue, and some of their jobs may run before some of yours, for fairness.
• You submit a job to a condo partition. The partition is currently full, so your job is queued. Other users from that department submit new jobs. As they are higher priority than you, their jobs run first.

You can view the currently running or queued jobs by running squeue on the head node, as described above. The NODELIST(REASON) column will indicate which jobs are queued by giving special codes:

• (Resources) means there are not enough resources (CPUs, memory, or GPUs) available for the job. It will run once the resources are free.
• (Priority) means resources are available, but Slurm wants to run a higher-priority job first. For example, maybe a large job has been waiting for resources for two days before you submit your job; Slurm may make your job wait so the large job can start first.

Job etiquette

The cluster is shared between users and departments. Therefore:

• Request the minimum resources required for your job, so other users can access other resources.
• If you need to submit a large number of jobs and aren't in a hurry, consider using your department's condo partition or all. Using all lets you get all available resources, but at lower priority, so other users can run jobs as well and you don't hog the cluster.
• Watch your disk space usage. With large deep models it is easy to use many terabytes of storage, and each department has a quota. You can check the quota and your own usage here.

Setting up jobs

Submitting batch jobs

The best way to run jobs is by writing a shell script specifying the Slurm options. To launch a batch job from a script you will need two files:

• the actual script in your favorite programming language that does whatever computation you want
• a bash script that requests the resources you need from Slurm and gives the commands to launch your job.

A typical bash script might look like this:

#!/bin/bash
# specify partition to run in (statds, phil, or cmist)
#SBATCH --partition statds
#SBATCH --ntasks 1
#SBATCH --job-name=nameofyourjob

# number of CPUs you need:
#SBATCH --cpus-per-task 1

# amount of memory per CPU
# (total allocated is CPUs * memory)
#SBATCH --mem-per-cpu 4G

# number of GPUs you need
#SBATCH --gres=gpu:1

# maximum time limit for job, HH:MM:SS (up to 48 hours max, 48:00:00)
#SBATCH --time=01:00:00

# email you when job is finished or errors out:
#SBATCH --mail-user=YourAndrewID@andrew.cmu.edu
#SBATCH --mail-type=END,FAIL

## Choose one:

# to run a Python script in a Conda environment called ENV
srun /home/export/YOURANDREWID/.conda/envs/ENV/bin/python YourScript.py

# to run an R script:
srun Rscript YourScript.R

Adjust the number of CPUs, number of GPUs, memory usage, and time limit based on the needs of your script. Requesting more than you need may result in your job having to wait for resources to free up; requesting a long time you don't need will block slurm from scheduling other jobs for when yours will be done. (And if you request more resources than exist on the compute nodes, your job will be queued forever while it waits for resources to appear.)

Note: Slurm batch jobs are limited to a maximum duration of 48 hours. Jobs running longer will be killed.

To run the job, SSH into the head node and navigate to the directory with your scripts. Then run:

sbatch example.sh

where example.sh is the name of your bash script.

You will see a message like "Submitted batch job 1209" when your job is submitted. Any printed output from your script will automatically be redirected to a file based on the job number -- in this case, slurm-1209.out. You can use tail -f slurm-1209.out to view the output live and monitor your script's progress.

Selecting the right resources

In the Slurm script above, you can choose the number of tasks, the number of GPUs, the memory per CPU, the number of GPUs, and the time limit. Choose these values wisely:

• You generally want 1 task (--ntasks 1). Asking for multiple tasks will cause srun to run multiple copies of your script simultaneously. If you need to run the same script with 10 different parameter values, this can be useful: request 10 tasks, and read the SLURM_PROCID environment variable in your script. It will be numbered 0-9 (for 10 tasks), so you can use this to set the parameter values for each parallel run.
• Set the number of CPUs based on your script. If your script is entirely single-threaded, ask for 1. If you use parallel processing (such as with multiprocessing in Python or the parallel package in R), use a multithreaded package, or spawn many parallel programs in your job, request more.
• Request the minimum amount of memory (RAM) needed for your script. This is per-CPU, so requesting a lot of CPUs and RAM can easily mean you request more RAM than exists on the node.
• Start with 1 GPU, adding more if your job requires more GPU memory (VRAM) than is on one GPU, or if you explicitly write your script to use more. If you do not use GPUs, request 0 so you aren't allocated a GPU that nobody else can use while your job runs.
• Set the time limit a little longer than you expect your script to run. Slurm uses the time limit when scheduling, so always choosing the maximum (48 hours) will backfire: Slurm will assume your job will take a long time and will run other things first. If you know your job can only take an hour, setting, say, a 01:30:00 time limit will let Slurm fit your job in before other long jobs in the queue. But of course if you choose too low and your job exceeds the limit, it will be killed, so leave yourself some extra time.

Interactive mode

You can also run interactive sessions that allow you to, say, run an R or Python console and type commands and immediately view the output. From the head node, run a command like:

# remember to edit to correct partition:
srun --partition statds --ntasks 1 \
    --cpus-per-task 16 --mem-per-cpu 4G \
    --gres=gpu:1 --time=01:00:00 --pty \
    bash

This runs a job that opens a bash shell on the compute node, and connects you directly to this job. You can now run bash commands, open Python, or run R scripts. Note your interactive session has a time limit and will be killed when it elapses, so set the time limit carefully and choose the resource amounts in the command according to your needs.

Interactive sessions are limited to two hours maximum. You cannot request a longer time limit. This is because interactive sessions are inefficient, with the cluster spending most of its time waiting for you to type something instead of running code. For large tasks, submit a batch job.

Interactive via VS Code

VS Code supports SSH and tunneling, which allows it to connect to a server and run commands. This allows you to use the Jupyter Notebook support built into VS Code to interact with the GPU cluster. As this uses an interactive session, it's subject to the same two-hour time limit. Hence notebooks should be used only for exploration and debugging, not large jobs.

To use this, we need to connect to the head node, run an interactive Slurm session, and then connect VS Code to that Slurm session so it can use it. This takes a few extra steps.

Setup

Before you use this for the first time, do the following steps:

• SSH into Wright and run the following to install and unpack the VSCode CLI: 

curl -Lk 'https://code.visualstudio.com/sha/download?build=stable&os=cli-alpine-x64' --output vscode_cli.tar.gz
tar -xf vscode_cli.tar.gz

• Add the code executable to your PATH by adding the following to your .bashrc file: export PATH="~:$PATH"

  echo 'export PATH="$PATH:/path/to/code"' >> ~/.bashrc

• In VSCode, install the Remote - Tunnels extension, then use the VSCode SSH feature to connect to Wright and make sure to install all the local VS Code extensions on the remote machine as well (e.g., Python, Jupyter, ...)

Running a session

• In the Wright headnode, launch a Slurm interactive session with srun, following the instructions above.

• Inside the interactive session, run code tunnel and follow the instructions. The first time the system will ask you to name the tunnel and authorize the process via an authenticator, I suggest using GitHub since it usually works quite smoothly. Subsequent tunnels might still ask you to authorize their creation by typing an alphanumerical sequence into a link that appears in the terminal. Once you’ve done that, the tunnel is successfully created.
• From VSCode, select Connect to Tunnel, then choose the authentication method you previously used and you will see the tunnel appear in the dropdown. Select it and you will land directly into the interactive session we just created
• You will now be able to run code and, e.g., Jupyter Notebooks using the compute node allocated by the Slurm interactive session. VS Code will be disconnected when the Slurm session ends, such as when it reaches its time limit.

Note that if you need to use a conda environment, you'll have to configure VS Code to use the Python executable you found above, since conda is not installed on the compute nodes.

SSH to Compute Nodes

The ability to SSH directly to compute nodes is disabled. You can run the command below to start a bash terminal on a compute node when running a batch job. This allows for the same functionality as SSH.

srun —jobid=yourjobidhere --pty /bin/bash