Wright GPU Cluster

From Stat DS Computing Wiki
Revision as of 18:13, 17 November 2025 by Jacobg2 (talk | contribs)

The department has access to a powerful GPU-based cluster named Wright which is hosted here at CMU. This is a 4-node and growing cluster that is a shared resource for Statistics & Data Science, Philosophy, and other departments in Dietrich College. Each department has priority access to its own compute node, but will be able to share the resources from other departments when not in use.

Access requests

This resource is free to use to users in the Departments of Statistics & Data Science and Philosophy, and in CMIST. These departments set policies determining which of their users are eligible. Users outside these departments must be sponsored by someone in the eligible departments, as part of a project or collaboration.

To request access, complete the access request form and our team will be in touch to discuss your request and onboard you.

Cluster setup

The cluster currently consists of 4 servers:

  • The head node: You log in to this to write code and run jobs. It has 128 GB of RAM, 16 cores, and no GPU.
  • The compute nodes: Jobs run on these. Each has 256 CPU cores, 8 NVIDIA L40 GPU cards with 48 GB of video RAM each, approximately 1 TB of RAM, and a total of approximately 3 TB of local data/scratch space.
  • The file server: Holds 50TB of storage. The head node and compute nodes connect via fiberoptic networking. Your home directory is hosted on the file server, so the same files are accessible on the head node and all compute nodes.

The cluster is currently partitioned. One compute node is reserved for each department involved.

Connecting and setting up your environment

SSH

From your terminal, type ssh <YourStatID>@wright.hss.cmu.edu . You can also avoid remembering this by adding the following snippet to your local ~/.ssh/config file: 

Host wright
    Hostname wright.hss.cmu.edu
    User YourStatID

You can then connect by simply typing ssh wright.

This connects you to the head node. Here you can upload your data files, set up your scripts, and test things.

Do not run resource-intensive tasks directly on the head node. It doesn't have any GPUs, and it has fewer resources than the compute nodes. You can do quick tests to make sure your scripts are set up correctly, but submit all other jobs through Slurm.

Uploading files

To upload scripts or data files, you can use scp from your local computer (Mac or Linux); on Windows, you can get the Tectia SSH graphical client free through CMU.

You can also connect through the Remote SSH feature of VS Code if you like. This is useful to 

  • navigate the file system from a proper GUI and 
  • modify code/files/folder in place if you are not familiar with the command line

Files uploaded to your home directory will be accessible on the head node and all compute nodes, so they can be freely used in compute jobs.

Using Conda

The head node has Anaconda installed for managing Python packages and system libraries; see Python virtual environments for information on how to use it (but note it's already installed, so you don't need to install it from scratch!).

However, the compute nodes do not have conda, so your slurm jobs can't use conda activate or other familiar commands. Instead, set up your environment on the head node, activate it, install the packages you need, and so on. Then run which python to get the path to the Python executable in your environment. For instance:

$ which python
/home/export/areinhar/.conda/envs/llm/bin/python

This is the path you'll use in your scripts (see below).

Running jobs

Slurm

The GPU cluster manages jobs via Slurm. Slurm is a job management system that deals with job prioritization and the allocation of resources. On high-performance computers (e.g., PSC and our GPU cluster) Slurm helps allocate resources so multiple jobs can be run quickly. Users run programs by submitting scripts to the Slurm job scheduler, and the scheduler runs them when resources are available.

To check what jobs are running, use squeue:

$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
              1326      phil     bash shaoanxi  R   22:41:00      1 n02
              1327      phil     bash shaoanxi  R   22:39:42      1 n02
              1486    statds run_coun  soheuny  R    1:12:53      1 n01
              1488    statds run_coun  soheuny  R    1:06:00      1 n01
              1489    statds run_coun  soheuny  R    1:00:53      1 n01
              1490    statds run_coun  soheuny  R    1:00:07      1 n01
              1491    statds run_coun  soheuny  R      24:02      1 n01
              1492    statds run_coun  soheuny  R       6:15      1 n01
              1493    statds run_coun  soheuny  R       3:16      1 n01
              1494    statds run_coun  soheuny  R       0:52      1 n01

Here we see there are eight jobs running on node 1 and 2 jobs running on node 2.

To check what resources are currently in use, use sinfo:

$ sinfo -O "NodeList:10,CPUsState:16,Memory:9,FreeMem:10,Gres:20,GresUsed:30"
NODELIST  CPUS(A/I/O/T)   MEMORY   FREE_MEM  GRES                GRES_USED                     
n01       64/192/0/256    1000000  221943    gpu:nvidia-l40:8    gpu:nvidia-l40:8(IDX:0-7)     
n02       32/224/0/256    1000000  225575    gpu:nvidia-l40:8    gpu:nvidia-l40:0(IDX:N/A)

This shows that node 1 (Statistics & Data Science) has 64 CPUs allocated and 192 idle, out of 256 total. There is 1TB of RAM, of which 222 GB is free. There are 8 NVIDIA L40 GPus, all 8 of which are in use. Hence if we submit a job to the statds partition requesting any GPUs, it will have to wait until some become free. Meanwhile, node 2 (Philosophy) has no GPUs in use.

Job queuing and scheduling

The cluster is divided into three partitions: statds (Statistics & Data Science), phil (Philosophy), and cmist (CMIST). Jobs are submitted to a partition. Users must submit jobs to their department's partition -- jobs submitted to the wrong partition will be rejected.

Slurm runs jobs based on the resources they request and what is available. If resources aren't available for your job, slurm will put it in the queue and wait until other jobs finish. However, it's not simply a first-in first-out queue -- Slurm attempts to arrange the jobs so every job can run, even if that means running them out of order.

Examples:

  • Someone submits Job A that requests 6 GPUs. Later, you submit Job B that requests 4 GPUs on the same partition. The compute node only has 8 GPUs, so Job B will be queued until Job A completes.
  • Someone submits Job A that requests 6 GPUs. Later, you submit Job B that requests 2 GPUs. Job B can run immediately.
  • You submit a job that requires all 8 GPUs on a node, but 4 are currently in use. Your job is put in the queue and scheduled to run when Slurm knows the current jobs will finish (because of their maximum time limit). Meanwhile, someone submits a job using 2 GPUs and a short time limit. Slurm lets it run immediately, because it knows it will be done before your job is scheduled to run.

You can view the currently running or queued jobs by running squeue on the head node, as described above. The NODELIST(REASON) column will indicate which jobs are queued by giving special codes:

  • (Resources) means there are not enough resources (CPUs, memory, or GPUs) available for the job. It will run once the resources are free.
  • (Priority) means resources are available, but Slurm wants to run a higher-priority job first. For example, maybe a large job has been waiting for resources for two days before you submit your job; Slurm may make your job wait so the large job can start first.

Submitting batch jobs

The best way to run jobs is by writing a shell script specifying the Slurm options. To launch a batch job from a script you will need two files:

  • the actual script that implements things in your favorite programming language
  • a bash script that requests the resources you need from Slurm and gives the commands to launch your job.

A typical bash script might look like this:

#!/bin/bash
# specify partition to run in (statds, phil, or cmist)
#SBATCH --partition statds
#SBATCH --ntasks 1
#SBATCH --job-name=nameofyourjob

# number of CPUs you need:
#SBATCH --cpus-per-task 1

# amount of memory per CPU
# (total allocated is CPUs * memory)
#SBATCH --mem-per-cpu 4G

# number of GPUs you need
#SBATCH --gres=gpu:1

# maximum time limit for task (up to 48 hours max, 48:00:00)
#SBATCH --time=01:00:00

# email you when job is finished or errors out:
#SBATCH --mail-user=YourAndrewID@andrew.cmu.edu
#SBATCH --mail-type=END,FAIL

## Choose one:

# to run a Python script in a Conda environment called ENV
srun /home/export/YOURANDREWID/.conda/envs/ENV/bin/python YourScript.py

# to run an R script:
srun Rscript YourScript.R

Adjust the number of CPUs, number of GPUs, memory usage, and time limit based on the needs of your script. Requesting more than you need may result in your job having to wait for resources to free up; requesting a long time you don't need will block slurm from scheduling other jobs for when yours will be done. (And if you request more resources than exist on the compute nodes, your job will be queued forever while it waits for resources to appear.)

Note: Slurm batch jobs are limited to a maximum duration of 48 hours. Jobs running longer will be killed.

To run the job, SSH into the head node and navigate to the directory with your scripts. Then run:

sbatch example.sh

where example.sh is the name of your bash script.

You will see a message like "Submitted batch job 1209" when your job is submitted. Any printed output from your script will automatically be redirected to a file based on the job number -- in this case, slurm-1209.out. You can use tail -f slurm-1209.out to view the output live and monitor your script's progress.

Interactive mode

You can also run interactive sessions that allow you to, say, run an R or Python console and type commands and immediately view the output. From the head node, run a command like:

# remember to edit to correct partition:
srun --partition statds --ntasks 1 \
    --cpus-per-task 16 --mem-per-cpu 4G \
    --gres=gpu:1 --time=01:00:00 --pty \
    bash

This runs a job that opens a bash shell on the compute node, and connects you directly to this job. You can now run bash commands, open Python, or run R scripts. Note your interactive session has a time limit and will be killed when it elapses, so set the time limit carefully and choose the resource amounts in the command according to your needs.

Interactive sessions are limited to two hours maximum. You cannot request a longer time limit. This is because interactive sessions are inefficient, with the cluster spending most of its time waiting for you to type something instead of running code. For large tasks, submit a batch job.

Interactive via VS Code

VS Code supports SSH and tunneling, which allows it to connect to a server and run commands. This allows you to use the Jupyter Notebook support built into VS Code to interact with the GPU cluster. As this uses an interactive session, it's subject to the same two-hour time limit. Hence notebooks should be used only for exploration and debugging, not large jobs.

To use this, we need to connect to the head node, run an interactive Slurm session, and then connect VS Code to that Slurm session so it can use it. This takes a few extra steps.

Setup

Before you use this for the first time, do the following steps:

  • SSH into Wright and run the following to install and unpack the VSCode CLI: 
curl -Lk 'https://code.visualstudio.com/sha/download?build=stable&os=cli-alpine-x64' --output vscode_cli.tar.gz
tar -xf vscode_cli.tar.gz
  • Add the code executable to your PATH by adding the following to your .bashrc file: export PATH="~:$PATH"
    echo 'export PATH="$PATH:/path/to/code"' >> ~/.bashrc
  • In VSCode, install the Remote - Tunnels extension, then use the VSCode SSH feature to connect to Wright and make sure to install all the local VS Code extensions on the remote machine as well (e.g., Python, Jupyter, ...)

Running a session

  • In the Wright headnode, launch a Slurm interactive session with srun, following the instructions above.
  • Inside the interactive session, run code tunnel and follow the instructions. The first time the system will ask you to name the tunnel and authorize the process via an authenticator, I suggest using GitHub since it usually works quite smoothly. Subsequent tunnels might still ask you to authorize their creation by typing an alphanumerical sequence into a link that appears in the terminal. Once you’ve done that, the tunnel is successfully created.
  • From VSCode, select Connect to Tunnel, then choose the authentication method you previously used and you will see the tunnel appear in the dropdown. Select it and you will land directly into the interactive session we just created
  • You will now be able to run code and, e.g., Jupyter Notebooks using the compute node allocated by the Slurm interactive session. VS Code will be disconnected when the Slurm session ends, such as when it reaches its time limit.

Note that if you need to use a conda environment, you'll have to configure VS Code to use the Python executable you found above, since conda is not installed on the compute nodes.

Submitting "condo" jobs

If your department's partition is busy, you may submit jobs to "condo" partitions to run on other nodes. The condo partitions are special:

  • Your job will run at a lower priority than that department's users. (Slurm calls this "QOS", so the jobs will have "normal" as their priority, vs. a "priority" value for the department's users.)
  • Your job will be preemptible: if the host department's users submit a job after yours that requires your resources, your job will be terminated and put back in the queue.

The condo partitions are named the same as the usual partitions, but ending with _condo. You must submit to the condo partition -- if you submit to another department's partition otherwise, your job will be rejected.

For example, if you are a Philosophy user, submitting a job to the statds_condo partition will run it on the Statistics node, but only when Statistics users are not using the resources. If a Statistics user submits a large job after yours, your job might be terminated, and will run again when resources become available.

SSH'ing to Nodes

The ability to SSH directly to compute nodes is disabled. You can run the command below to start a bash terminal on a compute node when running a batch job. This allows for the same functionality as SSH.

srun —jobid=yourjobidhere --pty /bin/bash
Retrieved from "https://info.stat.cmu.edu/index.php?title=Wright_GPU_Cluster&oldid=388"