Wright GPU Cluster

The department has access to a powerful GPU-based cluster named Wright which is hosted here at CMU. This is a 4-node and growing cluster that is a shared resource for Statistics & Data Science, Philosophy, and other departments in Dietrich College. Each department has priority access to its own compute node, but will be able to share the resources from other departments when not in use.

Access requests

This resource is free to use to users in the Departments of Statistics & Data Science, Philosophy, and CMIST. These departments set policies determining which of their users are eligible. Users outside these departments must be sponsored by someone in the eligible departments, as part of a project or collaboration.

To request access, complete the access request form and our team will be in touch to discuss your request and onboard you.

Cluster setup

The cluster currently consists of 4 servers:

• The head node: You log in to this to write code and run jobs. It has 128 GB of RAM, 16 cores, and no GPU.
• The compute nodes: Jobs run on these. Each has 256 CPU cores, 8 NVIDIA L40 GPU cards with 48 GB of video RAM each, approximately 1 TB of RAM, and a total of approximately 3 TB of local data/scratch space.
• The file server: Holds 50TB of storage. The head node and compute nodes connect via fiberoptic networking. Your home directory is hosted on the file server, so the same files are accessible on the head node and all compute nodes.
  • Statistics & Data Science accounts are limited to 5TB of space. More can be requested if required.

The cluster is currently partitioned. One compute node is reserved for each department involved.

The file server has quotas enabled so that each department gets 1/3rd of the available storage. If a department user tries to add files beyond the quota, the writes will be rejected, and nobody in that department will be able to add files until space is freed. To check the quotas and your own usage, view the quota report.

Connecting and setting up your environment

SSH

From your terminal, type ssh <YourStatID>@wright.hss.cmu.edu . You can also avoid remembering this by adding the following snippet to your local ~/.ssh/config file:

Host wright
    Hostname wright.hss.cmu.edu
    User YourStatID

You can then connect by simply typing ssh wright.

This connects you to the head node. Here you can upload your data files, set up your scripts, and test things.

Do not run resource-intensive tasks directly on the head node. It doesn't have any GPUs, and it has fewer resources than the compute nodes. You can do quick tests to make sure your scripts are set up correctly, but submit all other jobs through Slurm.

Uploading files

To upload scripts or data files, you can use scp from your local computer (Mac or Linux); on Windows, you can get the Tectia SSH graphical client free through CMU.

You can also connect through the Remote SSH feature of VS Code if you like. This is useful to

• navigate the file system from a proper GUI and 
• modify code/files/folder in place if you are not familiar with the command line

Files uploaded to your home directory will be accessible on the head node and all compute nodes, so they can be freely used in compute jobs.

Danger! Files are not backed up

Wright's storage cluster is not backed up. You may lose data if a disk fails. Therefore:

• Keep your code backed up in another location (such as by pushing it to GitHub regularly)
• Do not use Wright for long-term storage of important data. Download local copies, push it to HuggingFace, or sync it to your Google Drive
• Consider using an online backup tool like Borg or Tarsnap to regularly back up your home directory

Using Conda

The head node has Anaconda installed for managing Python packages and system libraries; see Python virtual environments for information on how to use it (but note it's already installed, so you don't need to install it from scratch!).

However, the compute nodes do not have conda, so your slurm jobs can't use conda activate or other familiar commands. Instead, set up your environment on the head node, activate it, install the packages you need, and so on. Then run which python to get the path to the Python executable in your environment. For instance:

$ which python
/home/export/areinhar/.conda/envs/llm/bin/python

This is the path you'll use in your scripts (see below).

Open OnDemand

Open OnDemand is a graphical interface where you can run jobs, interactive applications, and manage files. More information can be found here.

Job scheduling and prioritization

The GPU cluster manages jobs via Slurm. Slurm is a job management system that deals with job prioritization and the allocation of resources. On high-performance computers (e.g., PSC and our GPU cluster) Slurm helps allocate resources so multiple jobs can be run quickly. Users run programs by submitting scripts to the Slurm job scheduler, and the scheduler runs them when resources are available.

Queue status

To check what jobs are running, use squeue. The following options list more helpful information than the default (though you need a wide terminal to read it all):

$ squeue -O "JobID:7,UserName:10,Partition:12,TimeUsed:12,ReasonList,MinCPUS:9,MinMemory:12,tres-per-node,StartTime,EndTime"
JOBID  USER      PARTITION   TIME        NODELIST(REASON)    MIN_CPUS MIN_MEMORY  TRES_PER_NODE       START_TIME          END_TIME            
36229  ebensen   all         0:00        (DependencyNeverSati16       128G        N/A                 N/A                 N/A                 
36237  ajshen    all         12:35:57    n02                 2        64G         gres:gpu:1          2026-01-26T21:55:23 2026-01-28T21:55:23 
36152  msezgin   all         1-02:32:04  n03                 1        64G         gres:gpu:1          2026-01-26T07:59:16 2026-01-28T07:59:16 
36132  tsukiant  cmist_condo 1-11:00:02  n03                 16       8G          gres:gpu:1          2026-01-25T23:31:18 2026-01-27T23:31:18 
36100  tsukiant  cmist_condo 1-17:32:39  n03                 16       8G          gres:gpu:1          2026-01-25T16:58:41 2026-01-27T16:58:41 
36090  tsukiant  cmist_condo 1-18:22:51  n03                 16       8G          gres:gpu:1          2026-01-25T16:08:29 2026-01-27T16:08:29 
36247  jwalches  statds      42:38       n01                 40       15G         gres:gpu:3          2026-01-27T09:48:42 2026-01-29T09:48:42 
36051  xiangchs  statds_condo7:05:24     n01                 64       128G        gres:gpu:4          2026-01-27T03:25:56 2026-01-29T03:25:56

Here we see there are two jobs running on node 1, one on node 2, four on node 3, and one job that is not running because it was submitted with requirements that will not be met. The listed end times are based on the time limit specified when the job was submitted, so the jobs may end sooner. If you submit a job that is waiting for resources, the reason will be listed as (Resources) or (Priority), and after a few minutes Slurm will list a start time for your job based on when other jobs will finish and free up resources.

To check what resources are currently in use, use sinfo:

$ sinfo -O "NodeList:10,CPUsState:16,Memory:9,FreeMem:10,Gres:20,GresUsed:30"
NODELIST  CPUS(A/I/O/T)   MEMORY   FREE_MEM  GRES                GRES_USED                     
n01       64/192/0/256    1000000  221943    gpu:nvidia-l40:8    gpu:nvidia-l40:8(IDX:0-7)     
n02       32/224/0/256    1000000  225575    gpu:nvidia-l40:8    gpu:nvidia-l40:0(IDX:N/A)

This shows that node 1 (Statistics & Data Science) has 64 CPUs allocated and 192 idle, out of 256 total. There is 1TB of RAM, of which 222 GB is free. There are 8 NVIDIA L40 GPus, all 8 of which are in use. Hence if we submit a job to the statds partition requesting any GPUs, it will have to wait until some become free. Meanwhile, node 2 (Philosophy) has no GPUs in use.

Partitions

The cluster is divided into three main partitions:

• statds (Statistics & Data Science)
• phil (Philosophy)
• cmist (CMIST)

Jobs are submitted to a partition. The main partitions will only accept jobs from users in the corresponding department -- jobs submitted to the wrong partition will be rejected. Partitions correspond to nodes, so this ensures each department's jobs use their own node.

If you want to use other nodes, you can submit jobs to the all partition. This requests any available node from any partition, so your job will run on whichever node has resources available. However, your job will have lower priority, as described below; if your all job runs on the philosophy node, a user who submits to phil can preempt your job, causing it to be killed to make room for them, then restarted after their job is done. See Job Etiquette below for when to use all.

If you want to use a specific node within all you can specify the node within your submit script as shown below.

#SBATCH --nodelist=n03

Job prioritization

Slurm runs jobs based on the resources they request and what is available. If resources aren't available for your job, slurm will put it in the queue and wait until other jobs finish. However, it's not simply a first-in first-out queue -- Slurm attempts to arrange the jobs so every job can run, even if that means running them out of order. Partitions have different quality-of-service (QOS) values that affect their prioritization, and Slurm also adjusts each user's priority based on how many jobs they submit, so that one user can't hog the cluster by submitting dozens of jobs; if one user has many jobs waiting, theirs will be reduced in priority so other users can run jobs as well.

Examples:

• Someone submits Job A that requests 6 GPUs. Later, you submit Job B that requests 4 GPUs on the same partition. The compute node only has 8 GPUs, so Job B will be queued until Job A completes.
• Someone submits Job A that requests 6 GPUs. Later, you submit Job B that requests 2 GPUs. Job B can run immediately.
• You submit a job that requires all 8 GPUs on a node, but 4 are currently in use. Your job is put in the queue and scheduled to run when Slurm knows the current jobs will finish (because of their maximum time limit). Meanwhile, someone submits a job using 2 GPUs and a short time limit. Slurm lets it run immediately, because it knows it will be done before your job is scheduled to run. Other jobs may be put off until after yours runs, so yours can get the 8 GPUs it needs.
• You submit 50 jobs that each require 1 GPU. The node has only 8 GPUs, but several are in use. Several of your jobs run, but the others are queued for later. The fair-sharing system assigns them a lower priority, because you have run so many jobs, and jobs submitted by other users will run first.
• You submit Job A to the all partition. Someone else submits Job B to statds. The all partition has lower QOS, so Job B will get to use the StatDS node first, and Job A will either wait or use a different node.
• You submit Job A to the all partition. It begins to run on the philosophy node. Someone else submits Job B to phil, and there are insufficient resources to run both at the same time. Job A is killed, Job B runs, and Job A is placed back into the queue to be run when resources are available.

You can view the currently running or queued jobs by running squeue on the head node, as described above. The NODELIST(REASON) column will indicate which jobs are queued by giving special codes:

• (Resources) means there are not enough resources (CPUs, memory, or GPUs) available for the job. It will run once the resources are free.
• (Priority) means resources are available, but Slurm wants to run a higher-priority job first. For example, maybe a large job has been waiting for resources for two days before you submit your job; Slurm may make your job wait so the large job can start first.

Job etiquette

The cluster is shared between users and departments. Therefore:

• Request the minimum resources required for your job, so other users can access other resources.
• For ordinary one-off jobs, use your department's partition so you get top priority for your node.
• If you need to submit a large number of jobs and aren't in a hurry, consider using the all partition. Using all lets you get all available resources, but at lower priority, so other users can run jobs as well and you don't hog the cluster.
• Watch your disk space usage. With large deep models it is easy to use many terabytes of storage, and each department has a quota. You can check the quota and your own usage here.

Job limits

There is no practical limit to the number of jobs you can have in the queue at a time (there is a maximum size limit for job arrays that can be checked via scontrol show config | grep MaxArraySize. However, in order to prevent resource monopolization, no user can have more than 6 jobs running at once on the statds partition. If a user submits more than 6 jobs to the statds partition, the excess jobs will sit in the queue until one or more of the first 6 are completed. This limit does not apply to the all partition, so we encourage users with a large number of jobs to use the all partition instead of the statds partition.

Setting up jobs

Submitting batch jobs

The best way to run jobs is by writing a shell script specifying the Slurm options. To launch a batch job from a script you will need two files:

• the actual script in your favorite programming language that does whatever computation you want
• a bash script that requests the resources you need from Slurm and gives the commands to launch your job.

A typical bash script might look like this:

#!/bin/bash
# specify partition to run in (statds, phil, or cmist)
#SBATCH --partition statds
#SBATCH --ntasks 1
#SBATCH --job-name=nameofyourjob

# number of CPUs you need:
#SBATCH --cpus-per-task 1

# amount of memory per CPU
# (total allocated is CPUs * memory)
#SBATCH --mem-per-cpu 4G

# number of GPUs you need
#SBATCH --gres=gpu:1

# maximum time limit for job, HH:MM:SS (up to 48 hours max, 48:00:00)
#SBATCH --time=01:00:00

# email you when job is finished or errors out:
#SBATCH --mail-user=YourAndrewID@andrew.cmu.edu
#SBATCH --mail-type=END,FAIL

## Choose one:

# to run a Python script in a Conda environment called ENV
srun /home/export/YOURANDREWID/.conda/envs/ENV/bin/python YourScript.py

# to run an R script:
srun Rscript YourScript.R

Adjust the number of CPUs, number of GPUs, memory usage, and time limit based on the needs of your script. Requesting more than you need may result in your job having to wait for resources to free up; requesting a long time you don't need will block slurm from scheduling other jobs for when yours will be done. (And if you request more resources than exist on the compute nodes, your job will be queued forever while it waits for resources to appear.)

Note: Slurm batch jobs are limited to a maximum duration of 48 hours. Jobs running longer will be killed.

To run the job, SSH into the head node and navigate to the directory with your scripts. Then run:

sbatch example.sh

where example.sh is the name of your bash script.

You will see a message like "Submitted batch job 1209" when your job is submitted. Any printed output from your script will automatically be redirected to a file based on the job number -- in this case, slurm-1209.out. You can use tail -f slurm-1209.out to view the output live and monitor your script's progress.

Selecting the right resources

In the Slurm script above, you can choose the number of tasks, the number of GPUs, the memory per CPU, the number of GPUs, and the time limit. Choose these values wisely:

• You generally want 1 task (--ntasks 1). Asking for multiple tasks will cause srun to run multiple copies of your script simultaneously. If you need to run the same script with 10 different parameter values, this can be useful: request 10 tasks, and read the SLURM_PROCID environment variable in your script. It will be numbered 0-9 (for 10 tasks), so you can use this to set the parameter values for each parallel run.
• Set the number of CPUs based on your script. If your script is entirely single-threaded, ask for 1. If you use parallel processing (such as with multiprocessing in Python or the parallel package in R), use a multithreaded package, or spawn many parallel programs in your job, request more.
• Request the minimum amount of memory (RAM) needed for your script. This is per-CPU, so requesting a lot of CPUs and RAM can easily mean you request more RAM than exists on the node.
• Start with 1 GPU, adding more if your job requires more GPU memory (VRAM) than is on one GPU, or if you explicitly write your script to use more. If you do not use GPUs, request 0 so you aren't allocated a GPU that nobody else can use while your job runs.
• Set the time limit a little longer than you expect your script to run. Slurm uses the time limit when scheduling, so always choosing the maximum (48 hours) will backfire: Slurm will assume your job will take a long time and will run other things first. If you know your job can only take an hour, setting, say, a 01:30:00 time limit will let Slurm fit your job in before other long jobs in the queue. But of course if you choose too low and your job exceeds the limit, it will be killed, so leave yourself some extra time.

Interactive mode

You can also run interactive sessions that allow you to, say, run an R or Python console and type commands and immediately view the output. From the head node, run a command like:

# remember to edit to correct partition:
srun --partition statds --ntasks 1 \
    --cpus-per-task 16 --mem-per-cpu 4G \
    --gres=gpu:1 --time=01:00:00 --pty \
    bash

This runs a job that opens a bash shell on the compute node, and connects you directly to this job. You can now run bash commands, open Python, or run R scripts. Note your interactive session has a time limit and will be killed when it elapses, so set the time limit carefully and choose the resource amounts in the command according to your needs.

Interactive sessions are limited to two hours maximum. You cannot request a longer time limit. This is because interactive sessions are inefficient, with the cluster spending most of its time waiting for you to type something instead of running code. For large tasks, submit a batch job.

Interactive via VS Code

VS Code supports SSH and tunneling, which allows it to connect to a server and run commands. This allows you to use the Jupyter Notebook support built into VS Code to interact with the GPU cluster. As this uses an interactive session, it's subject to the same two-hour time limit. Hence notebooks should be used only for exploration and debugging, not large jobs.

To use this, we need to connect to the head node, run an interactive Slurm session, and then connect VS Code to that Slurm session so it can use it. This takes a few extra steps.

Setup

Before you use this for the first time, do the following steps:

• SSH into Wright and run the following to install and unpack the VSCode CLI: 

curl -Lk 'https://code.visualstudio.com/sha/download?build=stable&os=cli-alpine-x64' --output vscode_cli.tar.gz
tar -xf vscode_cli.tar.gz

• Add the code executable to your PATH by adding the following to your .bashrc file: export PATH="~:$PATH"

  echo 'export PATH="$PATH:/path/to/code"' >> ~/.bashrc

• In VSCode, install the Remote - Tunnels extension, then use the VSCode SSH feature to connect to Wright and make sure to install all the local VS Code extensions on the remote machine as well (e.g., Python, Jupyter, ...)

Running a session

• In the Wright headnode, launch a Slurm interactive session with srun, following the instructions above.

• Inside the interactive session, run code tunnel and follow the instructions. The first time the system will ask you to name the tunnel and authorize the process via an authenticator, I suggest using GitHub since it usually works quite smoothly. Subsequent tunnels might still ask you to authorize their creation by typing an alphanumerical sequence into a link that appears in the terminal. Once you’ve done that, the tunnel is successfully created.
• From VSCode, select Connect to Tunnel, then choose the authentication method you previously used and you will see the tunnel appear in the dropdown. Select it and you will land directly into the interactive session we just created
• You will now be able to run code and, e.g., Jupyter Notebooks using the compute node allocated by the Slurm interactive session. VS Code will be disconnected when the Slurm session ends, such as when it reaches its time limit.

Note that if you need to use a conda environment, you'll have to configure VS Code to use the Python executable you found above, since conda is not installed on the compute nodes.

SSH to Compute Nodes

The ability to SSH directly to compute nodes is disabled. You can run the command below to start a bash terminal on a compute node when running a batch job. This allows for the same functionality as SSH.

srun --jobid=yourjobidhere --pty /bin/bash

Monitoring cluster usage

The NVIDIA Wright GPU Metrics Dashboard shows graphs of GPU usage across the cluster. At the top left, you can select which node to graph, so viewing your department's node will let you see how GPUs are being used.

If you're developing a long-running job, we recommend checking this dashboard while your job is running. It is easy to accidentally write PyTorch or Tensorflow code that doesn't actually use the GPU, so check the graphs to see if the expected number of GPUs are in used based on your job's requirements and the other jobs listed in squeue. If they are not, get a terminal on the compute node as described above and run nvidia-smi to get GPU usage information, including which scripts are accessing the GPU. If your script doesn't show up, you can debug it to figure out why no GPUs are being used.

Using scratch storage

Each compute node (but not the head node) has several terabytes of scratch space. This is fast SSD storage that is directly attached to the node, rather than being accessed over the network (like all files in /home are). This is useful if your jobs need to write large temporary files: reading and writing the scratch space is faster than reading and writing files in /home.

Your home directory contains a symlink to the scratch space at ~/scratch. To use it, create a directory inside named after your username:

mkdir ~/scratch/yourusername

Then you can read and write files in that directory.

Warnings:

• The scratch space is meant for temporary storage during a job. It may be cleared out during cluster maintenance or if the disk is full.
• The scratch drives are much smaller than the huge networked disk used for /home, so be careful what you store in it, and regularly delete files you don't need.
• Each node has separate scratch disks, so data you write on one node is not available from any other node, or from the head node.

The scratch space is thus not for final model outputs or results. It is ideal for things like saving checkpoints while your job is running, dumping data to debug a job, saving output from one script to be immediately used by another script, etc.

Citing Wright

If you use Wright in your research, please make sure to cite it with the text provided below...

"This research was supported through the computational resources and staff contributions to the Wright HPC GPU cluster at Carnegie Mellon's Dietrich College of Humanities and Social Sciences provisioned and supported by the Department of Statistics & Data Science, the Department of Philosophy, The Carnegie Mellon Institute for Strategy and Technology, and the Dietrich College Dean's Office."