Wright GPU Cluster

The department has access to a powerful GPU-based cluster named Wright which is hosted here at CMU. This is a 3-node and growing cluster that is a shared resource with other departments in Dietrich. We have priority access to our own compute node, but will be able to share the resources from other departments when not in use.

Curious about the hardware?

The cluster currently consists of 4 servers:

• The headnode: You log in to this to write code and run jobs. It has 128 GB of RAM, 16 cores, and no GPU.
• The compute nodes: Jobs run on these. Each has 256 CPU cores, 8 NVIDIA L40 GPU cards, approximately 1 TB of RAM, and a total of approximately 3 TB of local data/scratch space.
• The fileserver: The headnode and compute nodes connect to this via fiberoptic networking for high-speed networking and has 50 TB of storage (N.B. this is not shared with hydras!)

ACCESS REQUEST FORM

This resource is free to use to everyone in the department.  As it is brand new and we’re still building our expertise (and documentation) and working out quirks, we do need to do a little gatekeeping.  If you anticipate that this is a necessary resource for you, please complete the following form and our team will be in touch to discuss your request and onboard you.

Connecting

SSH

From your terminal, type ssh <YourStatID>@wright.hss.cmu.edu . You can also avoid remembering this by adding the following snippet to your local .ssh/config file:

Host wright
    Hostname wright.hss.cmu.edu
    User YourStatID

You can then connect by simply typing ssh wright.

This connects you to the head node. Here you can upload your data files, set up your scripts, and test things.

Do not run resource-intensive tasks directly on the head node. It doesn't have any GPUs, and it has fewer resources than the compute nodes. You can do quick tests to make sure your scripts are set up correctly, but submit all other jobs through Slurm.

Uploading files

To upload scripts or data files, you can use scp from your local computer (Mac or Linux); on Windows, you can get the Tectia SSH graphical client free through CMU.

You can also connect through the Remote SSH feature of VS Code if you like. This is useful to 

• navigate the file system from a proper GUI and 
• modify code/files/folder in place if you are not familiar with the command line

Submitting jobs

Slurm

The GPU cluster manages jobs via Slurm. Slurm is a job management system that deals with job prioritization and the allocation of resources. On high-performance computers (e.g., PSC and our GPU cluster) Slurm helps make sure jobs are run as rapidly as possible. Users run programs by submitting scripts to the Slurm job scheduler. A Slurm script must do three things:

• prescribe the resource requirements for the job
• set the environment
• specify the work to be carried out in the form of shell commands

Note: At the moment, Slurm jobs on the GPU cluster are limited to a maximum duration of 48 hours.

Below are some more details about the two main modes of usage of Slurm: interactive and batch.

Batch Mode

This is the standard mode to launch jobs via a script. In the Wright headnode, to launch a batch job from a script you will need two files:

• the actual script that implements things in your favorite programming language
• a bash script that describes the Slurm settings and what to do to launch your job
• A typical bash script might look like this:

#!/bin/bash
#SBATCH --partition statds
#SBATCH --ntasks 1

# number of CPUs and amount of memory you need:
#SBATCH --cpus-per-task 16
#SBATCH --mem-per-cpu 16G

# number of GPUs you need
#SBATCH --gres=gpu:1

# maximum time limit for task (up to 48 hours, 48:00:00)
#SBATCH --time=01:00:00

# email you when job is finished:
#SBATCH --mail-user=YourAndrewID@andrew.cmu.edu
#SBATCH --mail-type=END

srun /home/export/YOURANDREWID/.conda/envs/ENV/bin/python YourScript.py

Interactive Mode

Goal: launch a Slurm interactive session and access it via the VSCode Tunneling feature.

VSCode has two main remote server features: SSH and Tunnels. Problem: the SSH feature of VSCode causes to land on the headnode, which is just used to schedule jobs. What we would like to do is to launch an interactive session and land into that session directly from VSCode.

Steps:

• SSH into Wright and run the following to install and unpack the VSCode CLI: 

curl -Lk 'https://code.visualstudio.com/sha/download?build=stable&os=cli-alpine-x64' --output vscode_cli.tar.gz
tar -xf vscode_cli.tar.gz

• Add the code executable to your PATH by adding the following to your .bashrc file: export PATH="~:$PATH"
• In VSCode, install the Remote - Tunnels extension, then use the VSCode SSH feature to connect to Wright and make sure to install all the local extensions on the remote machine as well (e.g., Python, Jupyter, ...)
• In the Wright headnode, launch a Slurm interactive session. The following example launches a session on the statds partition, requiring 5 minutes of time to execute 1 task on a single node. There are several more options to customize jobs.

srun --partition statds --nodes 1 --ntasks 1 --time=00:05:00 --pty bash

• Type code tunnel and follow the instructions. The first time the system will ask you to name the tunnel and authorize the process via an authenticator, I suggest using Github since it usually works quite smoothly. Subsequent tunnels might still ask you to authorize their creation by typing an alphanumerical sequence into a link that appears in the terminal. Once you’ve done that, the tunnel is successfully created
• From VSCode, select Connect to Tunnel, then choose the authentication method you previously used and you will see the tunnel appear in the dropdown. Select it and you will land directly into the interactive session we just created
• You will now be able to run code and, e.g., Jupyter Notebooks using the compute node allocated by the Slurm interactive session