Stat DS Computing Wiki - User contributions [en]

Wright GPU Cluster

2026-10-12T18:12:15Z

Jacobg2: /* Citing Wright */

The department has access to a powerful GPU-based cluster named [[wikipedia:Sewall_Wright|Wright]] which is hosted here at CMU. This is a 4-node and growing cluster that is a shared resource for Statistics & Data Science, Philosophy, and other departments in Dietrich College. Each department has priority access to its own compute node, but will be able to share the resources from other departments when not in use.

== Access requests ==
This resource is free to use to users in the Departments of Statistics & Data Science, Philosophy, and CMIST. These departments set policies determining which of their users are eligible. Users outside these departments must be sponsored by someone in the eligible departments, as part of a project or collaboration.

To request access, complete the [https://docs.google.com/forms/d/e/1FAIpQLSftCsplE06whGocn62DhwC6NpDSgZ-DMmUacrQID_slIfg6sA/viewform access request form] and our team will be in touch to discuss your request and onboard you.

== Cluster setup ==
The cluster currently consists of 4 servers:
*The head node: You log in to this to write code and run jobs. It has 128 GB of RAM, 16 cores, and '''no GPU'''.
*The compute nodes: Jobs run on these. Each has 256 CPU cores, 8 NVIDIA L40 GPU cards with 48 GB of video RAM each, approximately 1 TB of RAM, and a total of approximately 3 TB of local data/scratch space.
*The file server: Holds 50TB of storage. The head node and compute nodes connect via fiberoptic networking. Your home directory is hosted on the file server, so the same files are accessible on the head node and all compute nodes.
**Statistics & Data Science accounts are limited to 5TB of space. More can be requested if required.
The cluster is currently partitioned. One compute node is reserved for each department involved.

The file server has quotas enabled so that each department gets 1/3rd of the available storage. If a department user tries to add files beyond the quota, the writes will be rejected, and nobody in that department will be able to add files until space is freed. To check the quotas and your own usage, [https://util.stat.cmu.edu/quotareport/ view the quota report].

== Connecting and setting up your environment ==

=== SSH ===

From your terminal, type <code>ssh <YourStatID>@wright.hss.cmu.edu</code> . You can also avoid remembering this by adding the following snippet to your local <code>~/.ssh/config</code> file:
<pre>Host wright
Hostname wright.hss.cmu.edu
User YourStatID
</pre>

You can then connect by simply typing <code>ssh wright</code>.

This connects you to the '''head node.''' Here you can upload your data files, set up your scripts, and test things.

'''Do not run resource-intensive tasks directly on the head node'''. It doesn't have any GPUs, and it has fewer resources than the compute nodes. You can do quick tests to make sure your scripts are set up correctly, but submit all other jobs through Slurm.

=== Uploading files ===
To upload scripts or data files, you can use <code>scp</code> from your local computer (Mac or Linux); on Windows, you can get the Tectia SSH graphical client [https://www.cmu.edu/computing/services/endpoint/software/access.html free through CMU].

You can also connect through the [https://code.visualstudio.com/docs/remote/ssh Remote SSH feature of VS Code] if you like. This is useful to 

*navigate the file system from a proper GUI and 
*modify code/files/folder in place if you are not familiar with the command line
Files uploaded to your home directory will be accessible on the head node and all compute nodes, so they can be freely used in compute jobs.

=== Using Conda ===
The head node has Anaconda installed for managing Python packages and system libraries; see [[Python virtual environments]] for information on how to use it (but note it's '''already installed''', so you don't need to install it from scratch!).

However, the compute nodes '''do not''' have conda, so your slurm jobs can't use <code>conda activate</code> or other familiar commands. Instead, set up your environment on the head node, activate it, install the packages you need, and so on. Then run <code>which python</code> to get the path to the Python executable in your environment. For instance:<syntaxhighlight lang="bash">
$ which python
/home/export/areinhar/.conda/envs/llm/bin/python
</syntaxhighlight>This is the path you'll use in your scripts (see below).

=== Open OnDemand ===

Open OnDemand is a graphical interface where you can run jobs, interactive applications, and manage files. More information can be found [[Open OnDemand Wright|here]].

== Job scheduling and prioritization ==

The GPU cluster manages jobs via [[Slurm]]. Slurm is a job management system that deals with job prioritization and the allocation of resources. On high-performance computers (e.g., PSC and our GPU cluster) Slurm helps allocate resources so multiple jobs can be run quickly. Users run programs by submitting scripts to the Slurm job scheduler, and the scheduler runs them when resources are available.
=== Queue status ===

To check what jobs are running, use <code>squeue</code>. The following options list more helpful information than the default (though you need a wide terminal to read it all):<syntaxhighlight lang="bash">
$ squeue -O "JobID:7,UserName:10,Partition:12,TimeUsed:12,ReasonList,MinCPUS:9,MinMemory:12,tres-per-node,StartTime,EndTime"
JOBID USER PARTITION TIME NODELIST(REASON) MIN_CPUS MIN_MEMORY TRES_PER_NODE START_TIME END_TIME
36229 ebensen all 0:00 (DependencyNeverSati16 128G N/A N/A N/A
36237 ajshen all 12:35:57 n02 2 64G gres:gpu:1 2026-01-26T21:55:23 2026-01-28T21:55:23
36152 msezgin all 1-02:32:04 n03 1 64G gres:gpu:1 2026-01-26T07:59:16 2026-01-28T07:59:16
36132 tsukiant cmist_condo 1-11:00:02 n03 16 8G gres:gpu:1 2026-01-25T23:31:18 2026-01-27T23:31:18
36100 tsukiant cmist_condo 1-17:32:39 n03 16 8G gres:gpu:1 2026-01-25T16:58:41 2026-01-27T16:58:41
36090 tsukiant cmist_condo 1-18:22:51 n03 16 8G gres:gpu:1 2026-01-25T16:08:29 2026-01-27T16:08:29
36247 jwalches statds 42:38 n01 40 15G gres:gpu:3 2026-01-27T09:48:42 2026-01-29T09:48:42
36051 xiangchs statds_condo7:05:24 n01 64 128G gres:gpu:4 2026-01-27T03:25:56 2026-01-29T03:25:56
</syntaxhighlight>

Here we see there are two jobs running on node 1, one on node 2, four on node 3, and one job that is not running because it was submitted with requirements that will not be met. The listed end times are based on the time limit specified when the job was submitted, so the jobs may end sooner. If you submit a job that is waiting for resources, the reason will be listed as <code>(Resources)</code> or <code>(Priority)</code>, and after a few minutes Slurm will list a start time for your job based on when other jobs will finish and free up resources.

To check what resources are currently in use, use <code>sinfo</code>:<syntaxhighlight lang="bash">
$ sinfo -O "NodeList:10,CPUsState:16,Memory:9,FreeMem:10,Gres:20,GresUsed:30"
NODELIST CPUS(A/I/O/T) MEMORY FREE_MEM GRES GRES_USED
n01 64/192/0/256 1000000 221943 gpu:nvidia-l40:8 gpu:nvidia-l40:8(IDX:0-7)
n02 32/224/0/256 1000000 225575 gpu:nvidia-l40:8 gpu:nvidia-l40:0(IDX:N/A)
</syntaxhighlight>This shows that node 1 (Statistics & Data Science) has 64 CPUs allocated and 192 idle, out of 256 total. There is 1TB of RAM, of which 222 GB is free. There are 8 NVIDIA L40 GPus, all 8 of which are in use. Hence if we submit a job to the statds partition requesting any GPUs, it will have to wait until some become free. Meanwhile, node 2 (Philosophy) has no GPUs in use.

=== Partitions ===
The cluster is divided into three main partitions:

* <code>statds</code> (Statistics & Data Science)
* <code>phil</code> (Philosophy)
* <code>cmist</code> (CMIST)

Jobs are submitted to a partition. '''The main partitions will only accept jobs from users in the corresponding department''' -- jobs submitted to the wrong partition will be rejected. Partitions correspond to nodes, so this ensures each department's jobs use their own node.

If you want to use other nodes, you can submit jobs to the <code>all</code> partition. This requests any available node from any partition, so your job will run on whichever node has resources available. However, your job will have lower priority, as described below; if your all job runs on the philosophy node, a user who submits to <code>phil</code> can preempt your job, causing it to be killed to make room for them, then restarted after their job is done. See Job Etiquette below for when to use <code>all</code>.

If you want to use a specific node within <code>all</code> you can specify the node within your submit script as shown below.<syntaxhighlight lang="batch">
#SBATCH --nodelist=n03
</syntaxhighlight>

=== Job prioritization ===
Slurm runs jobs based on the resources they request and what is available. If resources aren't available for your job, slurm will put it in the queue and wait until other jobs finish. However, it's not simply a first-in first-out queue -- Slurm attempts to arrange the jobs so every job can run, even if that means running them out of order. Partitions have different quality-of-service (QOS) values that affect their prioritization, and Slurm also adjusts each user's priority based on how many jobs they submit, so that one user can't hog the cluster by submitting dozens of jobs; if one user has many jobs waiting, theirs will be reduced in priority so other users can run jobs as well.

Examples:

* Someone submits Job A that requests 6 GPUs. Later, you submit Job B that requests 4 GPUs on the same partition. The compute node only has 8 GPUs, so Job B will be queued until Job A completes.
* Someone submits Job A that requests 6 GPUs. Later, you submit Job B that requests 2 GPUs. Job B can run immediately.
* You submit a job that requires all 8 GPUs on a node, but 4 are currently in use. Your job is put in the queue and scheduled to run when Slurm knows the current jobs will finish (because of their maximum time limit). Meanwhile, someone submits a job using 2 GPUs and a short time limit. Slurm lets it run immediately, because it knows it will be done before your job is scheduled to run. Other jobs may be put off until after yours runs, so yours can get the 8 GPUs it needs.
* You submit 50 jobs that each require 1 GPU. The node has only 8 GPUs, but several are in use. Several of your jobs run, but the others are queued for later. The fair-sharing system assigns them a lower priority, because you have run so many jobs, and jobs submitted by other users will run first.
* You submit Job A to the <code>all</code> partition. Someone else submits Job B to <code>statds</code>. The <code>all</code> partition has lower QOS, so Job B will get to use the StatDS node first, and Job A will either wait or use a different node.
* You submit Job A to the <code>all</code> partition. It begins to run on the philosophy node. Someone else submits Job B to <code>phil</code>, and there are insufficient resources to run both at the same time. Job A is killed, Job B runs, and Job A is placed back into the queue to be run when resources are available.
You can view the currently running or queued jobs by running <code>squeue</code> on the head node, as described above. The <code>NODELIST(REASON)</code> column will indicate which jobs are queued by giving special codes:

* <code>(Resources)</code> means there are not enough resources (CPUs, memory, or GPUs) available for the job. It will run once the resources are free.
* <code>(Priority)</code> means resources are available, but Slurm wants to run a higher-priority job first. For example, maybe a large job has been waiting for resources for two days before you submit your job; Slurm may make your job wait so the large job can start first.

=== Job etiquette ===
The cluster is shared between users and departments. Therefore:

* Request the minimum resources required for your job, so other users can access other resources.
* For ordinary one-off jobs, use your department's partition so you get top priority for your node.
* If you need to submit a large number of jobs and aren't in a hurry, consider using the <code>all</code> partition. Using <code>all</code> lets you get all available resources, but at lower priority, so other users can run jobs as well and you don't hog the cluster.
* Watch your disk space usage. With large deep models it is easy to use many terabytes of storage, and each department has a quota. [https://util.stat.cmu.edu/quotareport/ You can check the quota and your own usage here.]

=== Job limits ===
There is no practical limit to the number of jobs you can have in the queue at a time (there is a maximum size limit for job arrays that can be checked via <code>scontrol show config | grep MaxArraySize</code>. However, in order to prevent resource monopolization, '''no user can have more than 6 jobs running at once on the <code>statds</code> partition.''' If a user submits more than 6 jobs to the <code>statds</code> partition, the excess jobs will sit in the queue until one or more of the first 6 are completed. '''This limit does not apply to the <code>all</code> partition''', so we encourage users with a large number of jobs to use the <code>all</code> partition instead of the <code>statds</code> partition.

== Setting up jobs ==

=== Submitting batch jobs ===
The best way to run jobs is by writing a shell script specifying the Slurm options. To launch a batch job from a script you will need two files:

*the actual script in your favorite programming language that does whatever computation you want
*a bash script that requests the resources you need from Slurm and gives the commands to launch your job.
A typical bash script might look like this:<pre>#!/bin/bash
# specify partition to run in (statds, phil, or cmist)
#SBATCH --partition statds
#SBATCH --ntasks 1
#SBATCH --job-name=nameofyourjob

# number of CPUs you need:
#SBATCH --cpus-per-task 1

# amount of memory per CPU
# (total allocated is CPUs * memory)
#SBATCH --mem-per-cpu 4G

# number of GPUs you need
#SBATCH --gres=gpu:1

# maximum time limit for job, HH:MM:SS (up to 48 hours max, 48:00:00)
#SBATCH --time=01:00:00

# email you when job is finished or errors out:
#SBATCH --mail-user=YourAndrewID@andrew.cmu.edu
#SBATCH --mail-type=END,FAIL

## Choose one:

# to run a Python script in a Conda environment called ENV
srun /home/export/YOURANDREWID/.conda/envs/ENV/bin/python YourScript.py

# to run an R script:
srun Rscript YourScript.R
</pre>Adjust the number of CPUs, number of GPUs, memory usage, and time limit based on the needs of your script. Requesting more than you need may result in your job having to wait for resources to free up; requesting a long time you don't need will block slurm from scheduling other jobs for when yours will be done. (And if you request more resources than exist on the compute nodes, your job will be queued forever while it waits for resources to appear.)

'''Note:''' Slurm batch jobs are limited to a maximum duration of 48 hours. Jobs running longer will be killed.

To run the job, SSH into the head node and navigate to the directory with your scripts. Then run:<syntaxhighlight lang="bash">
sbatch example.sh
</syntaxhighlight>where <code>example.sh</code> is the name of your bash script.

You will see a message like "Submitted batch job 1209" when your job is submitted. Any printed output from your script will automatically be redirected to a file based on the job number -- in this case, <code>slurm-1209.out</code>. You can use <code>tail -f slurm-1209.out</code> to view the output live and monitor your script's progress.

=== Selecting the right resources ===
In the Slurm script above, you can choose the number of tasks, the number of GPUs, the memory per CPU, the number of GPUs, and the time limit. Choose these values wisely:

* You generally want '''1 task''' (<code>--ntasks 1</code>). Asking for multiple tasks will cause srun to run multiple copies of your script simultaneously. If you need to run the same script with 10 different parameter values, this can be useful: request 10 tasks, and read the <code>SLURM_PROCID</code> environment variable in your script. It will be numbered 0-9 (for 10 tasks), so you can use this to set the parameter values for each parallel run.
* Set the number of CPUs based on your script. If your script is entirely single-threaded, ask for 1. If you use parallel processing (such as with multiprocessing in Python or the parallel package in R), use a multithreaded package, or spawn many parallel programs in your job, request more.
* Request the minimum amount of memory (RAM) needed for your script. This is per-CPU, so requesting a lot of CPUs and RAM can easily mean you request more RAM than exists on the node.
* Start with 1 GPU, adding more if your job requires more GPU memory (VRAM) than is on one GPU, or if you explicitly write your script to use more. If you do not use GPUs, request 0 so you aren't allocated a GPU that nobody else can use while your job runs.
* Set the time limit a little longer than you expect your script to run. Slurm uses the time limit when scheduling, so always choosing the maximum (48 hours) will backfire: Slurm will assume your job will take a long time and will run other things first. If you know your job can only take an hour, setting, say, a 01:30:00 time limit will let Slurm fit your job in before other long jobs in the queue. But of course if you choose too low and your job exceeds the limit, it will be killed, so leave yourself some extra time.

=== Interactive mode ===

You can also run interactive sessions that allow you to, say, run an R or Python console and type commands and immediately view the output. From the head node, run a command like:<syntaxhighlight lang="bash">
# remember to edit to correct partition:
srun --partition statds --ntasks 1 \
--cpus-per-task 16 --mem-per-cpu 4G \
--gres=gpu:1 --time=01:00:00 --pty \
bash
</syntaxhighlight>This runs a job that opens a bash shell on the compute node, and connects you directly to this job. You can now run bash commands, open Python, or run R scripts. Note your interactive session has a time limit and will be killed when it elapses, so set the time limit carefully and choose the resource amounts in the command according to your needs.

Interactive sessions are limited to '''two hours maximum'''. You cannot request a longer time limit. This is because interactive sessions are inefficient, with the cluster spending most of its time waiting for you to type something instead of running code. For large tasks, submit a batch job.

=== Interactive via VS Code ===
VS Code supports SSH and tunneling, which allows it to connect to a server and run commands. This allows you to use the Jupyter Notebook support built into VS Code to interact with the GPU cluster. As this uses an interactive session, it's subject to the same '''two-hour time limit'''. Hence notebooks should be used only for exploration and debugging, not large jobs.

To use this, we need to connect to the head node, run an interactive Slurm session, and then connect VS Code to that Slurm session so it can use it. This takes a few extra steps.

==== Setup ====
Before you use this for the first time, do the following steps:

*SSH into Wright and run the following to install and unpack the VSCode CLI: 
<pre>curl -Lk 'https://code.visualstudio.com/sha/download?build=stable&os=cli-alpine-x64' --output vscode_cli.tar.gz
tar -xf vscode_cli.tar.gz
</pre>

*Add the <code>code</code> executable to your PATH by adding the following to your <code>.bashrc</code> file: <code>export PATH="~:$PATH"</code><pre>echo 'export PATH="$PATH:/path/to/code"' >> ~/.bashrc
</pre>
*In VSCode, install the <code>Remote - Tunnels</code> extension, then use the VSCode SSH feature to connect to Wright and make sure to install all the local VS Code extensions on the remote machine as well (e.g., Python, Jupyter, ...)

==== Running a session ====
*In the Wright headnode, launch a Slurm interactive session with <code>srun</code>, following the instructions above.

*Inside the interactive session, run <code>code tunnel</code> and follow the instructions. The first time the system will ask you to name the tunnel and authorize the process via an authenticator, I suggest using GitHub since it usually works quite smoothly. Subsequent tunnels might still ask you to authorize their creation by typing an alphanumerical sequence into a link that appears in the terminal. Once you’ve done that, the tunnel is successfully created.
*From VSCode, select <code>Connect to Tunnel</code>, then choose the authentication method you previously used and you will see the tunnel appear in the dropdown. Select it and you will land directly into the interactive session we just created
*You will now be able to run code and, e.g., Jupyter Notebooks using the compute node allocated by the Slurm interactive session. VS Code will be disconnected when the Slurm session ends, such as when it reaches its time limit.
Note that if you need to use a conda environment, you'll have to configure VS Code to use the Python executable you found above, since conda is not installed on the compute nodes.

=== SSH to Compute Nodes ===
The ability to SSH directly to compute nodes is disabled. You can run the command below to start a bash terminal on a compute node when running a batch job. This allows for the same functionality as SSH.<syntaxhighlight lang="bash">
srun --jobid=yourjobidhere --pty /bin/bash
</syntaxhighlight>

== Monitoring cluster usage ==
The [https://util.stat.cmu.edu/grafana/d/Oxed_c6Wz/nvidia-wright-gpu-metrics NVIDIA Wright GPU Metrics Dashboard] shows graphs of GPU usage across the cluster. At the top left, you can select which node to graph, so viewing your department's node will let you see how GPUs are being used.

If you're developing a long-running job, we recommend checking this dashboard while your job is running. It is easy to accidentally write PyTorch or Tensorflow code that '''doesn't actually use the GPU''', so check the graphs to see if the expected number of GPUs are in used based on your job's requirements and the other jobs listed in <code>squeue</code>. If they are not, get a terminal on the compute node as described above and run <code>nvidia-smi</code> to get GPU usage information, including which scripts are accessing the GPU. If your script doesn't show up, you can debug it to figure out why no GPUs are being used.

== Using scratch storage ==
Each compute node (but not the head node) has several terabytes of scratch space. This is fast SSD storage that is directly attached to the node, rather than being accessed over the network (like all files in <code>/home</code> are). This is useful if your jobs need to write large temporary files: reading and writing the scratch space is faster than reading and writing files in <code>/home</code>.

Your home directory contains a symlink to the scratch space at <code>~/scratch</code>. To use it, create a directory inside named after your username:
mkdir ~/scratch/yourusername
Then you can read and write files in that directory.

'''Warnings:'''

* The scratch space is meant for '''temporary''' storage during a job. It may be cleared out during cluster maintenance or if the disk is full.
* The scratch drives are much smaller than the huge networked disk used for <code>/home</code>, so be careful what you store in it, and regularly delete files you don't need.
* Each node has separate scratch disks, so data you write on one node is '''not''' available from any other node, or from the head node.

The scratch space is thus not for final model outputs or results. It is ideal for things like saving checkpoints while your job is running, dumping data to debug a job, saving output from one script to be immediately used by another script, etc.

== Citing Wright ==
If you use Wright in your research, please make sure to cite it with the text provided below...<syntaxhighlight lang="text">
"This research was supported through the computational resources and staff contributions to the Wright HPC GPU cluster at Carnegie Mellon's Dietrich College of Humanities and Social Sciences provisioned and supported by the Department of Statistics & Data Science, the Department of Philosophy, The Carnegie Mellon Institute for Strategy and Technology, and the Dietrich College Dean's Office."
</syntaxhighlight>

Wright GPU Cluster

2026-10-12T18:11:05Z

Jacobg2: /* Citing Wright */

The department has access to a powerful GPU-based cluster named [[wikipedia:Sewall_Wright|Wright]] which is hosted here at CMU. This is a 4-node and growing cluster that is a shared resource for Statistics & Data Science, Philosophy, and other departments in Dietrich College. Each department has priority access to its own compute node, but will be able to share the resources from other departments when not in use.

== Access requests ==
This resource is free to use to users in the Departments of Statistics & Data Science, Philosophy, and CMIST. These departments set policies determining which of their users are eligible. Users outside these departments must be sponsored by someone in the eligible departments, as part of a project or collaboration.

To request access, complete the [https://docs.google.com/forms/d/e/1FAIpQLSftCsplE06whGocn62DhwC6NpDSgZ-DMmUacrQID_slIfg6sA/viewform access request form] and our team will be in touch to discuss your request and onboard you.

== Cluster setup ==
The cluster currently consists of 4 servers:
*The head node: You log in to this to write code and run jobs. It has 128 GB of RAM, 16 cores, and '''no GPU'''.
*The compute nodes: Jobs run on these. Each has 256 CPU cores, 8 NVIDIA L40 GPU cards with 48 GB of video RAM each, approximately 1 TB of RAM, and a total of approximately 3 TB of local data/scratch space.
*The file server: Holds 50TB of storage. The head node and compute nodes connect via fiberoptic networking. Your home directory is hosted on the file server, so the same files are accessible on the head node and all compute nodes.
**Statistics & Data Science accounts are limited to 5TB of space. More can be requested if required.
The cluster is currently partitioned. One compute node is reserved for each department involved.

The file server has quotas enabled so that each department gets 1/3rd of the available storage. If a department user tries to add files beyond the quota, the writes will be rejected, and nobody in that department will be able to add files until space is freed. To check the quotas and your own usage, [https://util.stat.cmu.edu/quotareport/ view the quota report].

== Connecting and setting up your environment ==

=== SSH ===

From your terminal, type <code>ssh <YourStatID>@wright.hss.cmu.edu</code> . You can also avoid remembering this by adding the following snippet to your local <code>~/.ssh/config</code> file:
<pre>Host wright
Hostname wright.hss.cmu.edu
User YourStatID
</pre>

You can then connect by simply typing <code>ssh wright</code>.

This connects you to the '''head node.''' Here you can upload your data files, set up your scripts, and test things.

'''Do not run resource-intensive tasks directly on the head node'''. It doesn't have any GPUs, and it has fewer resources than the compute nodes. You can do quick tests to make sure your scripts are set up correctly, but submit all other jobs through Slurm.

=== Uploading files ===
To upload scripts or data files, you can use <code>scp</code> from your local computer (Mac or Linux); on Windows, you can get the Tectia SSH graphical client [https://www.cmu.edu/computing/services/endpoint/software/access.html free through CMU].

You can also connect through the [https://code.visualstudio.com/docs/remote/ssh Remote SSH feature of VS Code] if you like. This is useful to 

*navigate the file system from a proper GUI and 
*modify code/files/folder in place if you are not familiar with the command line
Files uploaded to your home directory will be accessible on the head node and all compute nodes, so they can be freely used in compute jobs.

=== Using Conda ===
The head node has Anaconda installed for managing Python packages and system libraries; see [[Python virtual environments]] for information on how to use it (but note it's '''already installed''', so you don't need to install it from scratch!).

However, the compute nodes '''do not''' have conda, so your slurm jobs can't use <code>conda activate</code> or other familiar commands. Instead, set up your environment on the head node, activate it, install the packages you need, and so on. Then run <code>which python</code> to get the path to the Python executable in your environment. For instance:<syntaxhighlight lang="bash">
$ which python
/home/export/areinhar/.conda/envs/llm/bin/python
</syntaxhighlight>This is the path you'll use in your scripts (see below).

=== Open OnDemand ===

Open OnDemand is a graphical interface where you can run jobs, interactive applications, and manage files. More information can be found [[Open OnDemand Wright|here]].

== Job scheduling and prioritization ==

The GPU cluster manages jobs via [[Slurm]]. Slurm is a job management system that deals with job prioritization and the allocation of resources. On high-performance computers (e.g., PSC and our GPU cluster) Slurm helps allocate resources so multiple jobs can be run quickly. Users run programs by submitting scripts to the Slurm job scheduler, and the scheduler runs them when resources are available.
=== Queue status ===

To check what jobs are running, use <code>squeue</code>. The following options list more helpful information than the default (though you need a wide terminal to read it all):<syntaxhighlight lang="bash">
$ squeue -O "JobID:7,UserName:10,Partition:12,TimeUsed:12,ReasonList,MinCPUS:9,MinMemory:12,tres-per-node,StartTime,EndTime"
JOBID USER PARTITION TIME NODELIST(REASON) MIN_CPUS MIN_MEMORY TRES_PER_NODE START_TIME END_TIME
36229 ebensen all 0:00 (DependencyNeverSati16 128G N/A N/A N/A
36237 ajshen all 12:35:57 n02 2 64G gres:gpu:1 2026-01-26T21:55:23 2026-01-28T21:55:23
36152 msezgin all 1-02:32:04 n03 1 64G gres:gpu:1 2026-01-26T07:59:16 2026-01-28T07:59:16
36132 tsukiant cmist_condo 1-11:00:02 n03 16 8G gres:gpu:1 2026-01-25T23:31:18 2026-01-27T23:31:18
36100 tsukiant cmist_condo 1-17:32:39 n03 16 8G gres:gpu:1 2026-01-25T16:58:41 2026-01-27T16:58:41
36090 tsukiant cmist_condo 1-18:22:51 n03 16 8G gres:gpu:1 2026-01-25T16:08:29 2026-01-27T16:08:29
36247 jwalches statds 42:38 n01 40 15G gres:gpu:3 2026-01-27T09:48:42 2026-01-29T09:48:42
36051 xiangchs statds_condo7:05:24 n01 64 128G gres:gpu:4 2026-01-27T03:25:56 2026-01-29T03:25:56
</syntaxhighlight>

Here we see there are two jobs running on node 1, one on node 2, four on node 3, and one job that is not running because it was submitted with requirements that will not be met. The listed end times are based on the time limit specified when the job was submitted, so the jobs may end sooner. If you submit a job that is waiting for resources, the reason will be listed as <code>(Resources)</code> or <code>(Priority)</code>, and after a few minutes Slurm will list a start time for your job based on when other jobs will finish and free up resources.

To check what resources are currently in use, use <code>sinfo</code>:<syntaxhighlight lang="bash">
$ sinfo -O "NodeList:10,CPUsState:16,Memory:9,FreeMem:10,Gres:20,GresUsed:30"
NODELIST CPUS(A/I/O/T) MEMORY FREE_MEM GRES GRES_USED
n01 64/192/0/256 1000000 221943 gpu:nvidia-l40:8 gpu:nvidia-l40:8(IDX:0-7)
n02 32/224/0/256 1000000 225575 gpu:nvidia-l40:8 gpu:nvidia-l40:0(IDX:N/A)
</syntaxhighlight>This shows that node 1 (Statistics & Data Science) has 64 CPUs allocated and 192 idle, out of 256 total. There is 1TB of RAM, of which 222 GB is free. There are 8 NVIDIA L40 GPus, all 8 of which are in use. Hence if we submit a job to the statds partition requesting any GPUs, it will have to wait until some become free. Meanwhile, node 2 (Philosophy) has no GPUs in use.

=== Partitions ===
The cluster is divided into three main partitions:

* <code>statds</code> (Statistics & Data Science)
* <code>phil</code> (Philosophy)
* <code>cmist</code> (CMIST)

Jobs are submitted to a partition. '''The main partitions will only accept jobs from users in the corresponding department''' -- jobs submitted to the wrong partition will be rejected. Partitions correspond to nodes, so this ensures each department's jobs use their own node.

If you want to use other nodes, you can submit jobs to the <code>all</code> partition. This requests any available node from any partition, so your job will run on whichever node has resources available. However, your job will have lower priority, as described below; if your all job runs on the philosophy node, a user who submits to <code>phil</code> can preempt your job, causing it to be killed to make room for them, then restarted after their job is done. See Job Etiquette below for when to use <code>all</code>.

If you want to use a specific node within <code>all</code> you can specify the node within your submit script as shown below.<syntaxhighlight lang="batch">
#SBATCH --nodelist=n03
</syntaxhighlight>

=== Job prioritization ===
Slurm runs jobs based on the resources they request and what is available. If resources aren't available for your job, slurm will put it in the queue and wait until other jobs finish. However, it's not simply a first-in first-out queue -- Slurm attempts to arrange the jobs so every job can run, even if that means running them out of order. Partitions have different quality-of-service (QOS) values that affect their prioritization, and Slurm also adjusts each user's priority based on how many jobs they submit, so that one user can't hog the cluster by submitting dozens of jobs; if one user has many jobs waiting, theirs will be reduced in priority so other users can run jobs as well.

Examples:

* Someone submits Job A that requests 6 GPUs. Later, you submit Job B that requests 4 GPUs on the same partition. The compute node only has 8 GPUs, so Job B will be queued until Job A completes.
* Someone submits Job A that requests 6 GPUs. Later, you submit Job B that requests 2 GPUs. Job B can run immediately.
* You submit a job that requires all 8 GPUs on a node, but 4 are currently in use. Your job is put in the queue and scheduled to run when Slurm knows the current jobs will finish (because of their maximum time limit). Meanwhile, someone submits a job using 2 GPUs and a short time limit. Slurm lets it run immediately, because it knows it will be done before your job is scheduled to run. Other jobs may be put off until after yours runs, so yours can get the 8 GPUs it needs.
* You submit 50 jobs that each require 1 GPU. The node has only 8 GPUs, but several are in use. Several of your jobs run, but the others are queued for later. The fair-sharing system assigns them a lower priority, because you have run so many jobs, and jobs submitted by other users will run first.
* You submit Job A to the <code>all</code> partition. Someone else submits Job B to <code>statds</code>. The <code>all</code> partition has lower QOS, so Job B will get to use the StatDS node first, and Job A will either wait or use a different node.
* You submit Job A to the <code>all</code> partition. It begins to run on the philosophy node. Someone else submits Job B to <code>phil</code>, and there are insufficient resources to run both at the same time. Job A is killed, Job B runs, and Job A is placed back into the queue to be run when resources are available.
You can view the currently running or queued jobs by running <code>squeue</code> on the head node, as described above. The <code>NODELIST(REASON)</code> column will indicate which jobs are queued by giving special codes:

* <code>(Resources)</code> means there are not enough resources (CPUs, memory, or GPUs) available for the job. It will run once the resources are free.
* <code>(Priority)</code> means resources are available, but Slurm wants to run a higher-priority job first. For example, maybe a large job has been waiting for resources for two days before you submit your job; Slurm may make your job wait so the large job can start first.

=== Job etiquette ===
The cluster is shared between users and departments. Therefore:

* Request the minimum resources required for your job, so other users can access other resources.
* For ordinary one-off jobs, use your department's partition so you get top priority for your node.
* If you need to submit a large number of jobs and aren't in a hurry, consider using the <code>all</code> partition. Using <code>all</code> lets you get all available resources, but at lower priority, so other users can run jobs as well and you don't hog the cluster.
* Watch your disk space usage. With large deep models it is easy to use many terabytes of storage, and each department has a quota. [https://util.stat.cmu.edu/quotareport/ You can check the quota and your own usage here.]

=== Job limits ===
There is no practical limit to the number of jobs you can have in the queue at a time (there is a maximum size limit for job arrays that can be checked via <code>scontrol show config | grep MaxArraySize</code>. However, in order to prevent resource monopolization, '''no user can have more than 6 jobs running at once on the <code>statds</code> partition.''' If a user submits more than 6 jobs to the <code>statds</code> partition, the excess jobs will sit in the queue until one or more of the first 6 are completed. '''This limit does not apply to the <code>all</code> partition''', so we encourage users with a large number of jobs to use the <code>all</code> partition instead of the <code>statds</code> partition.

== Setting up jobs ==

=== Submitting batch jobs ===
The best way to run jobs is by writing a shell script specifying the Slurm options. To launch a batch job from a script you will need two files:

*the actual script in your favorite programming language that does whatever computation you want
*a bash script that requests the resources you need from Slurm and gives the commands to launch your job.
A typical bash script might look like this:<pre>#!/bin/bash
# specify partition to run in (statds, phil, or cmist)
#SBATCH --partition statds
#SBATCH --ntasks 1
#SBATCH --job-name=nameofyourjob

# number of CPUs you need:
#SBATCH --cpus-per-task 1

# amount of memory per CPU
# (total allocated is CPUs * memory)
#SBATCH --mem-per-cpu 4G

# number of GPUs you need
#SBATCH --gres=gpu:1

# maximum time limit for job, HH:MM:SS (up to 48 hours max, 48:00:00)
#SBATCH --time=01:00:00

# email you when job is finished or errors out:
#SBATCH --mail-user=YourAndrewID@andrew.cmu.edu
#SBATCH --mail-type=END,FAIL

## Choose one:

# to run a Python script in a Conda environment called ENV
srun /home/export/YOURANDREWID/.conda/envs/ENV/bin/python YourScript.py

# to run an R script:
srun Rscript YourScript.R
</pre>Adjust the number of CPUs, number of GPUs, memory usage, and time limit based on the needs of your script. Requesting more than you need may result in your job having to wait for resources to free up; requesting a long time you don't need will block slurm from scheduling other jobs for when yours will be done. (And if you request more resources than exist on the compute nodes, your job will be queued forever while it waits for resources to appear.)

'''Note:''' Slurm batch jobs are limited to a maximum duration of 48 hours. Jobs running longer will be killed.

To run the job, SSH into the head node and navigate to the directory with your scripts. Then run:<syntaxhighlight lang="bash">
sbatch example.sh
</syntaxhighlight>where <code>example.sh</code> is the name of your bash script.

You will see a message like "Submitted batch job 1209" when your job is submitted. Any printed output from your script will automatically be redirected to a file based on the job number -- in this case, <code>slurm-1209.out</code>. You can use <code>tail -f slurm-1209.out</code> to view the output live and monitor your script's progress.

=== Selecting the right resources ===
In the Slurm script above, you can choose the number of tasks, the number of GPUs, the memory per CPU, the number of GPUs, and the time limit. Choose these values wisely:

* You generally want '''1 task''' (<code>--ntasks 1</code>). Asking for multiple tasks will cause srun to run multiple copies of your script simultaneously. If you need to run the same script with 10 different parameter values, this can be useful: request 10 tasks, and read the <code>SLURM_PROCID</code> environment variable in your script. It will be numbered 0-9 (for 10 tasks), so you can use this to set the parameter values for each parallel run.
* Set the number of CPUs based on your script. If your script is entirely single-threaded, ask for 1. If you use parallel processing (such as with multiprocessing in Python or the parallel package in R), use a multithreaded package, or spawn many parallel programs in your job, request more.
* Request the minimum amount of memory (RAM) needed for your script. This is per-CPU, so requesting a lot of CPUs and RAM can easily mean you request more RAM than exists on the node.
* Start with 1 GPU, adding more if your job requires more GPU memory (VRAM) than is on one GPU, or if you explicitly write your script to use more. If you do not use GPUs, request 0 so you aren't allocated a GPU that nobody else can use while your job runs.
* Set the time limit a little longer than you expect your script to run. Slurm uses the time limit when scheduling, so always choosing the maximum (48 hours) will backfire: Slurm will assume your job will take a long time and will run other things first. If you know your job can only take an hour, setting, say, a 01:30:00 time limit will let Slurm fit your job in before other long jobs in the queue. But of course if you choose too low and your job exceeds the limit, it will be killed, so leave yourself some extra time.

=== Interactive mode ===

You can also run interactive sessions that allow you to, say, run an R or Python console and type commands and immediately view the output. From the head node, run a command like:<syntaxhighlight lang="bash">
# remember to edit to correct partition:
srun --partition statds --ntasks 1 \
--cpus-per-task 16 --mem-per-cpu 4G \
--gres=gpu:1 --time=01:00:00 --pty \
bash
</syntaxhighlight>This runs a job that opens a bash shell on the compute node, and connects you directly to this job. You can now run bash commands, open Python, or run R scripts. Note your interactive session has a time limit and will be killed when it elapses, so set the time limit carefully and choose the resource amounts in the command according to your needs.

Interactive sessions are limited to '''two hours maximum'''. You cannot request a longer time limit. This is because interactive sessions are inefficient, with the cluster spending most of its time waiting for you to type something instead of running code. For large tasks, submit a batch job.

=== Interactive via VS Code ===
VS Code supports SSH and tunneling, which allows it to connect to a server and run commands. This allows you to use the Jupyter Notebook support built into VS Code to interact with the GPU cluster. As this uses an interactive session, it's subject to the same '''two-hour time limit'''. Hence notebooks should be used only for exploration and debugging, not large jobs.

To use this, we need to connect to the head node, run an interactive Slurm session, and then connect VS Code to that Slurm session so it can use it. This takes a few extra steps.

==== Setup ====
Before you use this for the first time, do the following steps:

*SSH into Wright and run the following to install and unpack the VSCode CLI: 
<pre>curl -Lk 'https://code.visualstudio.com/sha/download?build=stable&os=cli-alpine-x64' --output vscode_cli.tar.gz
tar -xf vscode_cli.tar.gz
</pre>

*Add the <code>code</code> executable to your PATH by adding the following to your <code>.bashrc</code> file: <code>export PATH="~:$PATH"</code><pre>echo 'export PATH="$PATH:/path/to/code"' >> ~/.bashrc
</pre>
*In VSCode, install the <code>Remote - Tunnels</code> extension, then use the VSCode SSH feature to connect to Wright and make sure to install all the local VS Code extensions on the remote machine as well (e.g., Python, Jupyter, ...)

==== Running a session ====
*In the Wright headnode, launch a Slurm interactive session with <code>srun</code>, following the instructions above.

*Inside the interactive session, run <code>code tunnel</code> and follow the instructions. The first time the system will ask you to name the tunnel and authorize the process via an authenticator, I suggest using GitHub since it usually works quite smoothly. Subsequent tunnels might still ask you to authorize their creation by typing an alphanumerical sequence into a link that appears in the terminal. Once you’ve done that, the tunnel is successfully created.
*From VSCode, select <code>Connect to Tunnel</code>, then choose the authentication method you previously used and you will see the tunnel appear in the dropdown. Select it and you will land directly into the interactive session we just created
*You will now be able to run code and, e.g., Jupyter Notebooks using the compute node allocated by the Slurm interactive session. VS Code will be disconnected when the Slurm session ends, such as when it reaches its time limit.
Note that if you need to use a conda environment, you'll have to configure VS Code to use the Python executable you found above, since conda is not installed on the compute nodes.

=== SSH to Compute Nodes ===
The ability to SSH directly to compute nodes is disabled. You can run the command below to start a bash terminal on a compute node when running a batch job. This allows for the same functionality as SSH.<syntaxhighlight lang="bash">
srun --jobid=yourjobidhere --pty /bin/bash
</syntaxhighlight>

== Monitoring cluster usage ==
The [https://util.stat.cmu.edu/grafana/d/Oxed_c6Wz/nvidia-wright-gpu-metrics NVIDIA Wright GPU Metrics Dashboard] shows graphs of GPU usage across the cluster. At the top left, you can select which node to graph, so viewing your department's node will let you see how GPUs are being used.

If you're developing a long-running job, we recommend checking this dashboard while your job is running. It is easy to accidentally write PyTorch or Tensorflow code that '''doesn't actually use the GPU''', so check the graphs to see if the expected number of GPUs are in used based on your job's requirements and the other jobs listed in <code>squeue</code>. If they are not, get a terminal on the compute node as described above and run <code>nvidia-smi</code> to get GPU usage information, including which scripts are accessing the GPU. If your script doesn't show up, you can debug it to figure out why no GPUs are being used.

== Using scratch storage ==
Each compute node (but not the head node) has several terabytes of scratch space. This is fast SSD storage that is directly attached to the node, rather than being accessed over the network (like all files in <code>/home</code> are). This is useful if your jobs need to write large temporary files: reading and writing the scratch space is faster than reading and writing files in <code>/home</code>.

Your home directory contains a symlink to the scratch space at <code>~/scratch</code>. To use it, create a directory inside named after your username:
mkdir ~/scratch/yourusername
Then you can read and write files in that directory.

'''Warnings:'''

* The scratch space is meant for '''temporary''' storage during a job. It may be cleared out during cluster maintenance or if the disk is full.
* The scratch drives are much smaller than the huge networked disk used for <code>/home</code>, so be careful what you store in it, and regularly delete files you don't need.
* Each node has separate scratch disks, so data you write on one node is '''not''' available from any other node, or from the head node.

The scratch space is thus not for final model outputs or results. It is ideal for things like saving checkpoints while your job is running, dumping data to debug a job, saving output from one script to be immediately used by another script, etc.

== Citing Wright ==
If you use Wright in your research, please consider citing it with the text provided below...<syntaxhighlight lang="text">
"This research was supported through the computational resources and staff contributions to the Wright HPC GPU cluster at Carnegie Mellon's Dietrich College of Humanities and Social Sciences provisioned and supported by the Department of Statistics & Data Science, the Department of Philosophy, The Carnegie Mellon Institute for Strategy and Technology, and the Dietrich College Dean's Office."
</syntaxhighlight>

Wright GPU Cluster

2026-10-12T18:10:16Z

Jacobg2:

Wright GPU Cluster

2026-10-12T18:08:59Z

Jacobg2:

Wright GPU Cluster

2026-10-12T13:12:11Z

Jacobg2: /* Cluster setup */

[[index.php?title=Category:Wright Cluster]]

The department has access to a powerful GPU-based cluster named [https://en.wikipedia.org/wiki/Sewall_Wright Wright] which is hosted here at CMU. This is a 4-node and growing cluster that is a shared resource for Statistics & Data Science, Philosophy, and other departments in Dietrich College. Each department has priority access to its own compute node, but will be able to share the resources from other departments when not in use.

== Access requests ==
This resource is free to use to users in the Departments of Statistics & Data Science, Philosophy, and CMIST. These departments set policies determining which of their users are eligible. Users outside these departments must be sponsored by someone in the eligible departments, as part of a project or collaboration.

To request access, complete the [https://docs.google.com/forms/d/e/1FAIpQLSftCsplE06whGocn62DhwC6NpDSgZ-DMmUacrQID_slIfg6sA/viewform access request form] and our team will be in touch to discuss your request and onboard you.

== Cluster setup ==
The cluster currently consists of 4 servers:
*The head node: You log in to this to write code and run jobs. It has 128 GB of RAM, 16 cores, and '''no GPU'''.
*The compute nodes: Jobs run on these. Each has 256 CPU cores, 8 NVIDIA L40 GPU cards with 48 GB of video RAM each, approximately 1 TB of RAM, and a total of approximately 3 TB of local data/scratch space.
*The file server: Holds 50TB of storage. The head node and compute nodes connect via fiberoptic networking. Your home directory is hosted on the file server, so the same files are accessible on the head node and all compute nodes.
**Statistics & Data Science accounts are limited to 5TB of space. More can be requested if required.
The cluster is currently partitioned. One compute node is reserved for each department involved.

The file server has quotas enabled so that each department gets 1/3rd of the available storage. If a department user tries to add files beyond the quota, the writes will be rejected, and nobody in that department will be able to add files until space is freed. To check the quotas and your own usage, [https://util.stat.cmu.edu/quotareport/ view the quota report].

== Connecting and setting up your environment ==

=== SSH ===

From your terminal, type <code>ssh <YourStatID>@wright.hss.cmu.edu</code> . You can also avoid remembering this by adding the following snippet to your local <code>~/.ssh/config</code> file:
<pre>Host wright
Hostname wright.hss.cmu.edu
User YourStatID
</pre>

You can then connect by simply typing <code>ssh wright</code>.

This connects you to the '''head node.''' Here you can upload your data files, set up your scripts, and test things.

'''Do not run resource-intensive tasks directly on the head node'''. It doesn't have any GPUs, and it has fewer resources than the compute nodes. You can do quick tests to make sure your scripts are set up correctly, but submit all other jobs through Slurm.

=== Uploading files ===
To upload scripts or data files, you can use <code>scp</code> from your local computer (Mac or Linux); on Windows, you can get the Tectia SSH graphical client [https://www.cmu.edu/computing/services/endpoint/software/access.html free through CMU].

You can also connect through the [https://code.visualstudio.com/docs/remote/ssh Remote SSH feature of VS Code] if you like. This is useful to 

*navigate the file system from a proper GUI and 
*modify code/files/folder in place if you are not familiar with the command line
Files uploaded to your home directory will be accessible on the head node and all compute nodes, so they can be freely used in compute jobs.

=== Using Conda ===
The head node has Anaconda installed for managing Python packages and system libraries; see [[Python virtual environments]] for information on how to use it (but note it's '''already installed''', so you don't need to install it from scratch!).

However, the compute nodes '''do not''' have conda, so your slurm jobs can't use <code>conda activate</code> or other familiar commands. Instead, set up your environment on the head node, activate it, install the packages you need, and so on. Then run <code>which python</code> to get the path to the Python executable in your environment. For instance:<syntaxhighlight lang="bash">
$ which python
/home/export/areinhar/.conda/envs/llm/bin/python
</syntaxhighlight>This is the path you'll use in your scripts (see below).

=== Open OnDemand ===

Open OnDemand is a graphical interface where you can run jobs, interactive applications, and manage files. More information can be found [[Open OnDemand Wright|here]].

== Job scheduling and prioritization ==

The GPU cluster manages jobs via [[Slurm]]. Slurm is a job management system that deals with job prioritization and the allocation of resources. On high-performance computers (e.g., PSC and our GPU cluster) Slurm helps allocate resources so multiple jobs can be run quickly. Users run programs by submitting scripts to the Slurm job scheduler, and the scheduler runs them when resources are available.
=== Queue status ===

To check what jobs are running, use <code>squeue</code>. The following options list more helpful information than the default (though you need a wide terminal to read it all):<syntaxhighlight lang="bash">
$ squeue -O "JobID:7,UserName:10,Partition:12,TimeUsed:12,ReasonList,MinCPUS:9,MinMemory:12,tres-per-node,StartTime,EndTime"
JOBID USER PARTITION TIME NODELIST(REASON) MIN_CPUS MIN_MEMORY TRES_PER_NODE START_TIME END_TIME
36229 ebensen all 0:00 (DependencyNeverSati16 128G N/A N/A N/A
36237 ajshen all 12:35:57 n02 2 64G gres:gpu:1 2026-01-26T21:55:23 2026-01-28T21:55:23
36152 msezgin all 1-02:32:04 n03 1 64G gres:gpu:1 2026-01-26T07:59:16 2026-01-28T07:59:16
36132 tsukiant cmist_condo 1-11:00:02 n03 16 8G gres:gpu:1 2026-01-25T23:31:18 2026-01-27T23:31:18
36100 tsukiant cmist_condo 1-17:32:39 n03 16 8G gres:gpu:1 2026-01-25T16:58:41 2026-01-27T16:58:41
36090 tsukiant cmist_condo 1-18:22:51 n03 16 8G gres:gpu:1 2026-01-25T16:08:29 2026-01-27T16:08:29
36247 jwalches statds 42:38 n01 40 15G gres:gpu:3 2026-01-27T09:48:42 2026-01-29T09:48:42
36051 xiangchs statds_condo7:05:24 n01 64 128G gres:gpu:4 2026-01-27T03:25:56 2026-01-29T03:25:56
</syntaxhighlight>

Here we see there are two jobs running on node 1, one on node 2, four on node 3, and one job that is not running because it was submitted with requirements that will not be met. The listed end times are based on the time limit specified when the job was submitted, so the jobs may end sooner. If you submit a job that is waiting for resources, the reason will be listed as <code>(Resources)</code> or <code>(Priority)</code>, and after a few minutes Slurm will list a start time for your job based on when other jobs will finish and free up resources.

To check what resources are currently in use, use <code>sinfo</code>:<syntaxhighlight lang="bash">
$ sinfo -O "NodeList:10,CPUsState:16,Memory:9,FreeMem:10,Gres:20,GresUsed:30"
NODELIST CPUS(A/I/O/T) MEMORY FREE_MEM GRES GRES_USED
n01 64/192/0/256 1000000 221943 gpu:nvidia-l40:8 gpu:nvidia-l40:8(IDX:0-7)
n02 32/224/0/256 1000000 225575 gpu:nvidia-l40:8 gpu:nvidia-l40:0(IDX:N/A)
</syntaxhighlight>This shows that node 1 (Statistics & Data Science) has 64 CPUs allocated and 192 idle, out of 256 total. There is 1TB of RAM, of which 222 GB is free. There are 8 NVIDIA L40 GPus, all 8 of which are in use. Hence if we submit a job to the statds partition requesting any GPUs, it will have to wait until some become free. Meanwhile, node 2 (Philosophy) has no GPUs in use.

=== Partitions ===
The cluster is divided into three main partitions:

* <code>statds</code> (Statistics & Data Science)
* <code>phil</code> (Philosophy)
* <code>cmist</code> (CMIST)

Jobs are submitted to a partition. '''The main partitions will only accept jobs from users in the corresponding department''' -- jobs submitted to the wrong partition will be rejected. Partitions correspond to nodes, so this ensures each department's jobs use their own node.

If you want to use other nodes, you can submit jobs to the <code>all</code> partition. This requests any available node from any partition, so your job will run on whichever node has resources available. However, your job will have lower priority, as described below; if your all job runs on the philosophy node, a user who submits to <code>phil</code> can preempt your job, causing it to be killed to make room for them, then restarted after their job is done. See Job Etiquette below for when to use <code>all</code>.

If you want to use a specific node within <code>all</code> you can specify the node within your submit script as shown below.<syntaxhighlight lang="batch">
#SBATCH --nodelist=n03
</syntaxhighlight>

=== Job prioritization ===
Slurm runs jobs based on the resources they request and what is available. If resources aren't available for your job, slurm will put it in the queue and wait until other jobs finish. However, it's not simply a first-in first-out queue -- Slurm attempts to arrange the jobs so every job can run, even if that means running them out of order. Partitions have different quality-of-service (QOS) values that affect their prioritization, and Slurm also adjusts each user's priority based on how many jobs they submit, so that one user can't hog the cluster by submitting dozens of jobs; if one user has many jobs waiting, theirs will be reduced in priority so other users can run jobs as well.

Examples:

* Someone submits Job A that requests 6 GPUs. Later, you submit Job B that requests 4 GPUs on the same partition. The compute node only has 8 GPUs, so Job B will be queued until Job A completes.
* Someone submits Job A that requests 6 GPUs. Later, you submit Job B that requests 2 GPUs. Job B can run immediately.
* You submit a job that requires all 8 GPUs on a node, but 4 are currently in use. Your job is put in the queue and scheduled to run when Slurm knows the current jobs will finish (because of their maximum time limit). Meanwhile, someone submits a job using 2 GPUs and a short time limit. Slurm lets it run immediately, because it knows it will be done before your job is scheduled to run. Other jobs may be put off until after yours runs, so yours can get the 8 GPUs it needs.
* You submit 50 jobs that each require 1 GPU. The node has only 8 GPUs, but several are in use. Several of your jobs run, but the others are queued for later. The fair-sharing system assigns them a lower priority, because you have run so many jobs, and jobs submitted by other users will run first.
* You submit Job A to the <code>all</code> partition. Someone else submits Job B to <code>statds</code>. The <code>all</code> partition has lower QOS, so Job B will get to use the StatDS node first, and Job A will either wait or use a different node.
* You submit Job A to the <code>all</code> partition. It begins to run on the philosophy node. Someone else submits Job B to <code>phil</code>, and there are insufficient resources to run both at the same time. Job A is killed, Job B runs, and Job A is placed back into the queue to be run when resources are available.
You can view the currently running or queued jobs by running <code>squeue</code> on the head node, as described above. The <code>NODELIST(REASON)</code> column will indicate which jobs are queued by giving special codes:

* <code>(Resources)</code> means there are not enough resources (CPUs, memory, or GPUs) available for the job. It will run once the resources are free.
* <code>(Priority)</code> means resources are available, but Slurm wants to run a higher-priority job first. For example, maybe a large job has been waiting for resources for two days before you submit your job; Slurm may make your job wait so the large job can start first.

=== Job etiquette ===
The cluster is shared between users and departments. Therefore:

* Request the minimum resources required for your job, so other users can access other resources.
* For ordinary one-off jobs, use your department's partition so you get top priority for your node.
* If you need to submit a large number of jobs and aren't in a hurry, consider using the <code>all</code> partition. Using <code>all</code> lets you get all available resources, but at lower priority, so other users can run jobs as well and you don't hog the cluster.
* Watch your disk space usage. With large deep models it is easy to use many terabytes of storage, and each department has a quota. [https://util.stat.cmu.edu/quotareport/ You can check the quota and your own usage here.]

=== Job limits ===
There is no practical limit to the number of jobs you can have in the queue at a time (there is a maximum size limit for job arrays that can be checked via <code>scontrol show config | grep MaxArraySize</code>. However, in order to prevent resource monopolization, '''no user can have more than 6 jobs running at once on the <code>statds</code> partition.''' If a user submits more than 6 jobs to the <code>statds</code> partition, the excess jobs will sit in the queue until one or more of the first 6 are completed. '''This limit does not apply to the <code>all</code> partition''', so we encourage users with a large number of jobs to use the <code>all</code> partition instead of the <code>statds</code> partition.

== Setting up jobs ==

=== Submitting batch jobs ===
The best way to run jobs is by writing a shell script specifying the Slurm options. To launch a batch job from a script you will need two files:

*the actual script in your favorite programming language that does whatever computation you want
*a bash script that requests the resources you need from Slurm and gives the commands to launch your job.
A typical bash script might look like this:<pre>#!/bin/bash
# specify partition to run in (statds, phil, or cmist)
#SBATCH --partition statds
#SBATCH --ntasks 1
#SBATCH --job-name=nameofyourjob

# number of CPUs you need:
#SBATCH --cpus-per-task 1

# amount of memory per CPU
# (total allocated is CPUs * memory)
#SBATCH --mem-per-cpu 4G

# number of GPUs you need
#SBATCH --gres=gpu:1

# maximum time limit for job, HH:MM:SS (up to 48 hours max, 48:00:00)
#SBATCH --time=01:00:00

# email you when job is finished or errors out:
#SBATCH --mail-user=YourAndrewID@andrew.cmu.edu
#SBATCH --mail-type=END,FAIL

## Choose one:

# to run a Python script in a Conda environment called ENV
srun /home/export/YOURANDREWID/.conda/envs/ENV/bin/python YourScript.py

# to run an R script:
srun Rscript YourScript.R
</pre>Adjust the number of CPUs, number of GPUs, memory usage, and time limit based on the needs of your script. Requesting more than you need may result in your job having to wait for resources to free up; requesting a long time you don't need will block slurm from scheduling other jobs for when yours will be done. (And if you request more resources than exist on the compute nodes, your job will be queued forever while it waits for resources to appear.)

'''Note:''' Slurm batch jobs are limited to a maximum duration of 48 hours. Jobs running longer will be killed.

To run the job, SSH into the head node and navigate to the directory with your scripts. Then run:<syntaxhighlight lang="bash">
sbatch example.sh
</syntaxhighlight>where <code>example.sh</code> is the name of your bash script.

You will see a message like "Submitted batch job 1209" when your job is submitted. Any printed output from your script will automatically be redirected to a file based on the job number -- in this case, <code>slurm-1209.out</code>. You can use <code>tail -f slurm-1209.out</code> to view the output live and monitor your script's progress.

=== Selecting the right resources ===
In the Slurm script above, you can choose the number of tasks, the number of GPUs, the memory per CPU, the number of GPUs, and the time limit. Choose these values wisely:

* You generally want '''1 task''' (<code>--ntasks 1</code>). Asking for multiple tasks will cause srun to run multiple copies of your script simultaneously. If you need to run the same script with 10 different parameter values, this can be useful: request 10 tasks, and read the <code>SLURM_PROCID</code> environment variable in your script. It will be numbered 0-9 (for 10 tasks), so you can use this to set the parameter values for each parallel run.
* Set the number of CPUs based on your script. If your script is entirely single-threaded, ask for 1. If you use parallel processing (such as with multiprocessing in Python or the parallel package in R), use a multithreaded package, or spawn many parallel programs in your job, request more.
* Request the minimum amount of memory (RAM) needed for your script. This is per-CPU, so requesting a lot of CPUs and RAM can easily mean you request more RAM than exists on the node.
* Start with 1 GPU, adding more if your job requires more GPU memory (VRAM) than is on one GPU, or if you explicitly write your script to use more. If you do not use GPUs, request 0 so you aren't allocated a GPU that nobody else can use while your job runs.
* Set the time limit a little longer than you expect your script to run. Slurm uses the time limit when scheduling, so always choosing the maximum (48 hours) will backfire: Slurm will assume your job will take a long time and will run other things first. If you know your job can only take an hour, setting, say, a 01:30:00 time limit will let Slurm fit your job in before other long jobs in the queue. But of course if you choose too low and your job exceeds the limit, it will be killed, so leave yourself some extra time.

=== Interactive mode ===

You can also run interactive sessions that allow you to, say, run an R or Python console and type commands and immediately view the output. From the head node, run a command like:<syntaxhighlight lang="bash">
# remember to edit to correct partition:
srun --partition statds --ntasks 1 \
--cpus-per-task 16 --mem-per-cpu 4G \
--gres=gpu:1 --time=01:00:00 --pty \
bash
</syntaxhighlight>This runs a job that opens a bash shell on the compute node, and connects you directly to this job. You can now run bash commands, open Python, or run R scripts. Note your interactive session has a time limit and will be killed when it elapses, so set the time limit carefully and choose the resource amounts in the command according to your needs.

Interactive sessions are limited to '''two hours maximum'''. You cannot request a longer time limit. This is because interactive sessions are inefficient, with the cluster spending most of its time waiting for you to type something instead of running code. For large tasks, submit a batch job.

=== Interactive via VS Code ===
VS Code supports SSH and tunneling, which allows it to connect to a server and run commands. This allows you to use the Jupyter Notebook support built into VS Code to interact with the GPU cluster. As this uses an interactive session, it's subject to the same '''two-hour time limit'''. Hence notebooks should be used only for exploration and debugging, not large jobs.

To use this, we need to connect to the head node, run an interactive Slurm session, and then connect VS Code to that Slurm session so it can use it. This takes a few extra steps.

==== Setup ====
Before you use this for the first time, do the following steps:

*SSH into Wright and run the following to install and unpack the VSCode CLI: 
<pre>curl -Lk 'https://code.visualstudio.com/sha/download?build=stable&os=cli-alpine-x64' --output vscode_cli.tar.gz
tar -xf vscode_cli.tar.gz
</pre>

*Add the <code>code</code> executable to your PATH by adding the following to your <code>.bashrc</code> file: <code>export PATH="~:$PATH"</code><pre>echo 'export PATH="$PATH:/path/to/code"' >> ~/.bashrc
</pre>
*In VSCode, install the <code>Remote - Tunnels</code> extension, then use the VSCode SSH feature to connect to Wright and make sure to install all the local VS Code extensions on the remote machine as well (e.g., Python, Jupyter, ...)

==== Running a session ====
*In the Wright headnode, launch a Slurm interactive session with <code>srun</code>, following the instructions above.

*Inside the interactive session, run <code>code tunnel</code> and follow the instructions. The first time the system will ask you to name the tunnel and authorize the process via an authenticator, I suggest using GitHub since it usually works quite smoothly. Subsequent tunnels might still ask you to authorize their creation by typing an alphanumerical sequence into a link that appears in the terminal. Once you’ve done that, the tunnel is successfully created.
*From VSCode, select <code>Connect to Tunnel</code>, then choose the authentication method you previously used and you will see the tunnel appear in the dropdown. Select it and you will land directly into the interactive session we just created
*You will now be able to run code and, e.g., Jupyter Notebooks using the compute node allocated by the Slurm interactive session. VS Code will be disconnected when the Slurm session ends, such as when it reaches its time limit.
Note that if you need to use a conda environment, you'll have to configure VS Code to use the Python executable you found above, since conda is not installed on the compute nodes.

=== SSH to Compute Nodes ===
The ability to SSH directly to compute nodes is disabled. You can run the command below to start a bash terminal on a compute node when running a batch job. This allows for the same functionality as SSH.<syntaxhighlight lang="bash">
srun --jobid=yourjobidhere --pty /bin/bash
</syntaxhighlight>

== Monitoring cluster usage ==
The [https://util.stat.cmu.edu/grafana/d/Oxed_c6Wz/nvidia-wright-gpu-metrics NVIDIA Wright GPU Metrics Dashboard] shows graphs of GPU usage across the cluster. At the top left, you can select which node to graph, so viewing your department's node will let you see how GPUs are being used.

If you're developing a long-running job, we recommend checking this dashboard while your job is running. It is easy to accidentally write PyTorch or Tensorflow code that '''doesn't actually use the GPU''', so check the graphs to see if the expected number of GPUs are in used based on your job's requirements and the other jobs listed in <code>squeue</code>. If they are not, get a terminal on the compute node as described above and run <code>nvidia-smi</code> to get GPU usage information, including which scripts are accessing the GPU. If your script doesn't show up, you can debug it to figure out why no GPUs are being used.

== Using scratch storage ==
Each compute node (but not the head node) has several terabytes of scratch space. This is fast SSD storage that is directly attached to the node, rather than being accessed over the network (like all files in <code>/home</code> are). This is useful if your jobs need to write large temporary files: reading and writing the scratch space is faster than reading and writing files in <code>/home</code>.

Your home directory contains a symlink to the scratch space at <code>~/scratch</code>. To use it, create a directory inside named after your username:
mkdir ~/scratch/yourusername
Then you can read and write files in that directory.

'''Warnings:'''

* The scratch space is meant for '''temporary''' storage during a job. It may be cleared out during cluster maintenance or if the disk is full.
* The scratch drives are much smaller than the huge networked disk used for <code>/home</code>, so be careful what you store in it, and regularly delete files you don't need.
* Each node has separate scratch disks, so data you write on one node is '''not''' available from any other node, or from the head node.

The scratch space is thus not for final model outputs or results. It is ideal for things like saving checkpoints while your job is running, dumping data to debug a job, saving output from one script to be immediately used by another script, etc.

CommentStreams:91d525b39f89ae7acf4eb6a1587f0ce1

2026-09-16T18:16:40Z

Jacobg2:

hi{{DISPLAYTITLE:
test
}}

Main Page

2026-08-12T12:16:03Z

Jacobg2:

Welcome to the CMU Department of Statistics and Data Science Computing Wiki! To get the most out of this resource, please note the following:

<div style="background-color: #e2e3e5; border: 5px solid #6c757d; padding: 15px; margin-bottom: 20px; margin-top: 15px;">
'''Purpose & Contribution'''<br>
The main purpose of this wiki is to familiarize readers with the computing resources provided by the department.
* '''Newcomers:''' Please read the [[Beginner's Guide to Computation in Statistical Research]] to familiarize yourself with statistical computing tools and best practices as you begin you research.
* '''Computational Researchers:''' See the [[Article Catalog]] for a full selection of topics and pages and the [[Directory of Computing Resources for Statistical Research]] for available computational resources. You can also use the search bar at the top of this page. For one-off questions, consider using the [https://util.stat.cmu.edu/statchat/ Support Chat]
* '''Contributors and Maintainers:''' Feel free to leave a comment on any page, or create your own pages. Please see [[Generic Wiki Information]] for more information.
</div>

== Important Announcements ==
Before using any department computing resources, please see [[Research Computing Dos and Don'ts]]<div style="background-color: #fff8e6; border-left: 4px solid #ffc107; padding: 15px; margin-bottom: 20px;">
* '''[2026-04-10] New Article:''' Introducing [[Globus]], a tool for sharing large files within and between institutions.
* '''[2026-04-01] PSA:''' If you use VS Code, look for "[https://marketplace.visualstudio.com/items?itemName=ms-python.vscode-python-envs Python Environments]" in your extensions and '''uninstall it immediately''' due to environment conflicts.
* '''[2026-04-01] Updated Article:''' The [[Linux and department servers]] page has been updated with instructions on how to use <code>/scratch</code> to avoid disk I/O slowdowns. See [[Linux and department servers#Using /scratch]].
</div>

== Quick Start ==
<div style="display: grid; grid-template-columns: repeat(auto-fit, minmax(500px, 1fr))">
<div style="background-color: #f8f9fa; border-left: 4px solid #005A9C; padding: 15px; margin-bottom: 10px;">
'''[[Beginner's Guide to Computation in Statistical Research]]'''<br>
The essential starting point for understanding how to scale your statistical workflows and choose the right tools for your research.
</div>

<div style="background-color: #f8f9fa; border-left: 4px solid #28a745; padding: 15px; margin-bottom: 10px;">
'''[[Directory of Computing Resources for Statistical Research]]'''<br>
A comprehensive index of all available clusters, servers, and computing environments accessible to department members.
</div>

<div style="background-color: #f8f9fa; border-left: 4px solid #6f42c1; padding: 15px; margin-bottom: 10px;">
'''[[Department Tools and Services]]'''<br>
Additional department tools, printing, web hosting, software licenses, and specialized computing services.
</div>

<div style="background-color: #f8f9fa; border-left: 4px solid #eff07d; padding: 15px; margin-bottom: 10px;">
'''[[Article Catalog]]'''<br>
A comprehensive list of articles organized by topic
</div>

</div>

== Support Resources ==
For assistance with issues not covered in this wiki, please reach out to the following contacts:

<div style="display: grid; grid-template-columns: repeat(auto-fit, minmax(500px, 1fr))">
<div style="background-color: #f8f9fa; border-left: 4px solid #6c757d; padding: 15px; margin-bottom: 10px;">
'''General Tech Support'''<br>
For general issues with department computing resources:<br>
'''Email:''' [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu]
</div>

<div style="background-color: #f8f9fa; border-left: 4px solid #17a2b8; padding: 15px; margin-bottom: 10px;">
'''Research Project Assistance (Graduate Students Only)'''<br>
Live assistance from Alex Shen, PhD Computing TA, for project-specific computing questions:<br>
'''Office Hours:''' Mondays 2:00 PM - 3:00 PM @ FMS Interview Room 320<br>
'''Contact:''' [mailto:ajshen@andrew.cmu.edu ajshen@andrew.cmu.edu] or reach out directly via Slack.
</div>

<div style="background-color: #f8f9fa; border-left: 4px solid #17a2b8; padding: 15px; margin-bottom: 10px;">
'''Onboarding & Troubleshooting (Graduate Students Only)'''<br>
Live assistance for setting up and troubleshooting computing resources:<br>
'''Carl Skipper:''' Office hours by appointment. Request via [mailto:skipper@andrew.cmu.edu skipper@andrew.cmu.edu]<br>
'''Jake Gordon:''' Office hours by appointment. Request via [mailto:jacobg2@andrew.cmu.edu jacobg2@andrew.cmu.edu]
</div>
</div>

== Suggestions Box and Todos ==

<div style="background-color: #aff0ff; border: 4px solid #17a2b8; padding: 15px; margin-bottom: 10px;">
* '''[New Article]''' Beginner's Guide -> Scaling -> Version Control -> Experiment Management
* '''[New Article]''' Beginner's Guide -> Scaling -> Big Data Management -> Articles about file formats and compression
* '''[New Article]''' Beginner's Guide -> Software Development -> Package Creation -> Writing Documentation
* '''[New Article]''' Beginner's Guide -> Software Development -> Package Repositories

</div>

This section contains work in progress and future suggestions for the wiki. If you would like to leave a suggestions, you may:
* Directly contact the Computing TA
* Leave a comment on a specific wiki article with your suggestions
* Leave a comment on this page with your suggestions
<br>

Main Page

2026-08-12T12:15:55Z

Jacobg2:

Welcome to the CMU Department of Statistics and Data Science Computing Wiki! To get the most out of this resource, please note the following: test

<div style="background-color: #e2e3e5; border: 5px solid #6c757d; padding: 15px; margin-bottom: 20px; margin-top: 15px;">
'''Purpose & Contribution'''<br>
The main purpose of this wiki is to familiarize readers with the computing resources provided by the department.
* '''Newcomers:''' Please read the [[Beginner's Guide to Computation in Statistical Research]] to familiarize yourself with statistical computing tools and best practices as you begin you research.
* '''Computational Researchers:''' See the [[Article Catalog]] for a full selection of topics and pages and the [[Directory of Computing Resources for Statistical Research]] for available computational resources. You can also use the search bar at the top of this page. For one-off questions, consider using the [https://util.stat.cmu.edu/statchat/ Support Chat]
* '''Contributors and Maintainers:''' Feel free to leave a comment on any page, or create your own pages. Please see [[Generic Wiki Information]] for more information.
</div>

== Important Announcements ==
Before using any department computing resources, please see [[Research Computing Dos and Don'ts]]<div style="background-color: #fff8e6; border-left: 4px solid #ffc107; padding: 15px; margin-bottom: 20px;">
* '''[2026-04-10] New Article:''' Introducing [[Globus]], a tool for sharing large files within and between institutions.
* '''[2026-04-01] PSA:''' If you use VS Code, look for "[https://marketplace.visualstudio.com/items?itemName=ms-python.vscode-python-envs Python Environments]" in your extensions and '''uninstall it immediately''' due to environment conflicts.
* '''[2026-04-01] Updated Article:''' The [[Linux and department servers]] page has been updated with instructions on how to use <code>/scratch</code> to avoid disk I/O slowdowns. See [[Linux and department servers#Using /scratch]].
</div>

== Quick Start ==
<div style="display: grid; grid-template-columns: repeat(auto-fit, minmax(500px, 1fr))">
<div style="background-color: #f8f9fa; border-left: 4px solid #005A9C; padding: 15px; margin-bottom: 10px;">
'''[[Beginner's Guide to Computation in Statistical Research]]'''<br>
The essential starting point for understanding how to scale your statistical workflows and choose the right tools for your research.
</div>

<div style="background-color: #f8f9fa; border-left: 4px solid #28a745; padding: 15px; margin-bottom: 10px;">
'''[[Directory of Computing Resources for Statistical Research]]'''<br>
A comprehensive index of all available clusters, servers, and computing environments accessible to department members.
</div>

<div style="background-color: #f8f9fa; border-left: 4px solid #6f42c1; padding: 15px; margin-bottom: 10px;">
'''[[Department Tools and Services]]'''<br>
Additional department tools, printing, web hosting, software licenses, and specialized computing services.
</div>

<div style="background-color: #f8f9fa; border-left: 4px solid #eff07d; padding: 15px; margin-bottom: 10px;">
'''[[Article Catalog]]'''<br>
A comprehensive list of articles organized by topic
</div>

</div>

== Support Resources ==
For assistance with issues not covered in this wiki, please reach out to the following contacts:

<div style="display: grid; grid-template-columns: repeat(auto-fit, minmax(500px, 1fr))">
<div style="background-color: #f8f9fa; border-left: 4px solid #6c757d; padding: 15px; margin-bottom: 10px;">
'''General Tech Support'''<br>
For general issues with department computing resources:<br>
'''Email:''' [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu]
</div>

<div style="background-color: #f8f9fa; border-left: 4px solid #17a2b8; padding: 15px; margin-bottom: 10px;">
'''Research Project Assistance (Graduate Students Only)'''<br>
Live assistance from Alex Shen, PhD Computing TA, for project-specific computing questions:<br>
'''Office Hours:''' Mondays 2:00 PM - 3:00 PM @ FMS Interview Room 320<br>
'''Contact:''' [mailto:ajshen@andrew.cmu.edu ajshen@andrew.cmu.edu] or reach out directly via Slack.
</div>

<div style="background-color: #f8f9fa; border-left: 4px solid #17a2b8; padding: 15px; margin-bottom: 10px;">
'''Onboarding & Troubleshooting (Graduate Students Only)'''<br>
Live assistance for setting up and troubleshooting computing resources:<br>
'''Carl Skipper:''' Office hours by appointment. Request via [mailto:skipper@andrew.cmu.edu skipper@andrew.cmu.edu]<br>
'''Jake Gordon:''' Office hours by appointment. Request via [mailto:jacobg2@andrew.cmu.edu jacobg2@andrew.cmu.edu]
</div>
</div>

== Suggestions Box and Todos ==

<div style="background-color: #aff0ff; border: 4px solid #17a2b8; padding: 15px; margin-bottom: 10px;">
* '''[New Article]''' Beginner's Guide -> Scaling -> Version Control -> Experiment Management
* '''[New Article]''' Beginner's Guide -> Scaling -> Big Data Management -> Articles about file formats and compression
* '''[New Article]''' Beginner's Guide -> Software Development -> Package Creation -> Writing Documentation
* '''[New Article]''' Beginner's Guide -> Software Development -> Package Repositories

</div>

This section contains work in progress and future suggestions for the wiki. If you would like to leave a suggestions, you may:
* Directly contact the Computing TA
* Leave a comment on a specific wiki article with your suggestions
* Leave a comment on this page with your suggestions
<br>

Main Page

2026-08-11T19:25:43Z

Jacobg2:

Main Page

2026-08-11T19:25:37Z

Jacobg2:

Welcome to the CMU Department of Statistics and Data Science Computing Wiki! To get the most out of this resource, please note the following: test1

<div style="background-color: #e2e3e5; border: 5px solid #6c757d; padding: 15px; margin-bottom: 20px; margin-top: 15px;">
'''Purpose & Contribution'''<br>
The main purpose of this wiki is to familiarize readers with the computing resources provided by the department.
* '''Newcomers:''' Please read the [[Beginner's Guide to Computation in Statistical Research]] to familiarize yourself with statistical computing tools and best practices as you begin you research.
* '''Computational Researchers:''' See the [[Article Catalog]] for a full selection of topics and pages and the [[Directory of Computing Resources for Statistical Research]] for available computational resources. You can also use the search bar at the top of this page. For one-off questions, consider using the [https://util.stat.cmu.edu/statchat/ Support Chat]
* '''Contributors and Maintainers:''' Feel free to leave a comment on any page, or create your own pages. Please see [[Generic Wiki Information]] for more information.
</div>

== Important Announcements ==
Before using any department computing resources, please see [[Research Computing Dos and Don'ts]]<div style="background-color: #fff8e6; border-left: 4px solid #ffc107; padding: 15px; margin-bottom: 20px;">
* '''[2026-04-10] New Article:''' Introducing [[Globus]], a tool for sharing large files within and between institutions.
* '''[2026-04-01] PSA:''' If you use VS Code, look for "[https://marketplace.visualstudio.com/items?itemName=ms-python.vscode-python-envs Python Environments]" in your extensions and '''uninstall it immediately''' due to environment conflicts.
* '''[2026-04-01] Updated Article:''' The [[Linux and department servers]] page has been updated with instructions on how to use <code>/scratch</code> to avoid disk I/O slowdowns. See [[Linux and department servers#Using /scratch]].
</div>

== Quick Start ==
<div style="display: grid; grid-template-columns: repeat(auto-fit, minmax(500px, 1fr))">
<div style="background-color: #f8f9fa; border-left: 4px solid #005A9C; padding: 15px; margin-bottom: 10px;">
'''[[Beginner's Guide to Computation in Statistical Research]]'''<br>
The essential starting point for understanding how to scale your statistical workflows and choose the right tools for your research.
</div>

<div style="background-color: #f8f9fa; border-left: 4px solid #28a745; padding: 15px; margin-bottom: 10px;">
'''[[Directory of Computing Resources for Statistical Research]]'''<br>
A comprehensive index of all available clusters, servers, and computing environments accessible to department members.
</div>

<div style="background-color: #f8f9fa; border-left: 4px solid #6f42c1; padding: 15px; margin-bottom: 10px;">
'''[[Department Tools and Services]]'''<br>
Additional department tools, printing, web hosting, software licenses, and specialized computing services.
</div>

<div style="background-color: #f8f9fa; border-left: 4px solid #eff07d; padding: 15px; margin-bottom: 10px;">
'''[[Article Catalog]]'''<br>
A comprehensive list of articles organized by topic
</div>

</div>

== Support Resources ==
For assistance with issues not covered in this wiki, please reach out to the following contacts:

<div style="display: grid; grid-template-columns: repeat(auto-fit, minmax(500px, 1fr))">
<div style="background-color: #f8f9fa; border-left: 4px solid #6c757d; padding: 15px; margin-bottom: 10px;">
'''General Tech Support'''<br>
For general issues with department computing resources:<br>
'''Email:''' [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu]
</div>

<div style="background-color: #f8f9fa; border-left: 4px solid #17a2b8; padding: 15px; margin-bottom: 10px;">
'''Research Project Assistance (Graduate Students Only)'''<br>
Live assistance from Alex Shen, PhD Computing TA, for project-specific computing questions:<br>
'''Office Hours:''' Mondays 2:00 PM - 3:00 PM @ FMS Interview Room 320<br>
'''Contact:''' [mailto:ajshen@andrew.cmu.edu ajshen@andrew.cmu.edu] or reach out directly via Slack.
</div>

<div style="background-color: #f8f9fa; border-left: 4px solid #17a2b8; padding: 15px; margin-bottom: 10px;">
'''Onboarding & Troubleshooting (Graduate Students Only)'''<br>
Live assistance for setting up and troubleshooting computing resources:<br>
'''Carl Skipper:''' Office hours by appointment. Request via [mailto:skipper@andrew.cmu.edu skipper@andrew.cmu.edu]<br>
'''Jake Gordon:''' Office hours by appointment. Request via [mailto:jacobg2@andrew.cmu.edu jacobg2@andrew.cmu.edu]
</div>
</div>

== Suggestions Box and Todos ==

<div style="background-color: #aff0ff; border: 4px solid #17a2b8; padding: 15px; margin-bottom: 10px;">
* '''[New Article]''' Beginner's Guide -> Scaling -> Version Control -> Experiment Management
* '''[New Article]''' Beginner's Guide -> Scaling -> Big Data Management -> Articles about file formats and compression
* '''[New Article]''' Beginner's Guide -> Software Development -> Package Creation -> Writing Documentation
* '''[New Article]''' Beginner's Guide -> Software Development -> Package Repositories

</div>

This section contains work in progress and future suggestions for the wiki. If you would like to leave a suggestions, you may:
* Directly contact the Computing TA
* Leave a comment on a specific wiki article with your suggestions
* Leave a comment on this page with your suggestions
<br>

Main Page

2026-08-11T19:19:49Z

Jacobg2:

Main Page

2026-08-11T19:19:40Z

Jacobg2:

Jupyter Notebooks

2026-08-11T19:10:49Z

Jacobg2:

[[Category:IDEs]]
[[Category:Remote Server Access]]
[[Category:Department Servers]]

Running Jupyter Notebooks is now possible with Open OnDemand. To run a Notebook, follow the steps below.

1 - Log in to '''Open OnDemand.'''

* https://ood.stat.cmu.edu/

2 - Click the '''Interactive Apps''' drop down menu and then select '''Jupyter Notebook.'''
[[File:OOD Jupyter Menu Bar.png|left|frameless|719x719px]]

3 - You will be taken to a new page where you can select options for the Notebook. Fill out the form as desired and then click the '''Launch''' button. Use [https://util.stat.cmu.edu/grafana/d/asjkldfbnvajdsfh213/leviathan-cluster?orgId=1&from=now-6h&to=now&timezone=browser Grafana] to help determine what server to use. Select one with low resource usage.

'''Hours''' - The number of hours your Notebook will run.

'''Server''' - The server to run your Notebook on.

'''Jupyter Notebook Path''' - The path the Notebook environment lives on. You can use the default path which has many statistics related Python packages preinstalled, or you can define your own Notebook environment. See below for details on how to set your own environment up.

[[File:Jupyter Notebook Page.png|frameless|403x403px]]

4 - You will be taken to another new page where the Jupyter Notebook will load. Once it's ready, click '''Connect to Jupyter''' at the bottom. You will then be taken to a new tab with your Notebook session.

[[File:NotebookSession.png|frameless|526x526px]]

=== Custom Jupyter Notebook Environment ===
To build a custom Notebook environment, you can use a Python package manager such as Momba or Python's built in virtual environments. This example will cover Python virtual environments.

# Declare the virtual Python environment.
#* ''python3 -m venv /path/to/your/jupyter_env''
# Activate the virtual environment.
#*''source /path/to/your/jupyter_env/bin/activate''
# Install the Jupyter Notebook application to the new Python virtual environment.
#* ''pip install notebook''

To use your own Jupyter Environment, replace the default path above in step 3 for Jupyter Notebook Path.

Jupyter Notebooks

2026-08-11T19:10:41Z

Jacobg2:

[[Category:IDEs]]
[[Category:Remote Server Access]]
[[Category:Department Servers]]

Running Jupyter Notebooks is now possible with Open OnDemand. To run a Notebook, follow the steps below.

1 - Log in to '''Open OnDemand.'''

* https://ood.stat.cmu.edu/

2 - Click the '''Interactive Apps''' drop down menu and then select '''Jupyter Notebook.'''
[[File:OOD Jupyter Menu Bar.png|left|frameless|719x719px]]

3 - You will be taken to a new page where you can select options for the Notebook. Fill out the form as desired and then click the '''Launch''' button. Use [https://util.stat.cmu.edu/grafana/d/asjkldfbnvajdsfh213/leviathan-cluster?orgId=1&from=now-6h&to=now&timezone=browser Grafana] to help determine what server to use. Select one with low resource usage.

'''Hours''' - The number of hours your Notebook will run.

'''Server''' - The server to run your Notebook on. test

'''Jupyter Notebook Path''' - The path the Notebook environment lives on. You can use the default path which has many statistics related Python packages preinstalled, or you can define your own Notebook environment. See below for details on how to set your own environment up.

[[File:Jupyter Notebook Page.png|frameless|403x403px]]

4 - You will be taken to another new page where the Jupyter Notebook will load. Once it's ready, click '''Connect to Jupyter''' at the bottom. You will then be taken to a new tab with your Notebook session.

[[File:NotebookSession.png|frameless|526x526px]]

=== Custom Jupyter Notebook Environment ===
To build a custom Notebook environment, you can use a Python package manager such as Momba or Python's built in virtual environments. This example will cover Python virtual environments.

# Declare the virtual Python environment.
#* ''python3 -m venv /path/to/your/jupyter_env''
# Activate the virtual environment.
#*''source /path/to/your/jupyter_env/bin/activate''
# Install the Jupyter Notebook application to the new Python virtual environment.
#* ''pip install notebook''

To use your own Jupyter Environment, replace the default path above in step 3 for Jupyter Notebook Path.

Jupyter Notebooks

2026-08-11T19:10:33Z

Jacobg2:

[[Category:IDEs]]
[[Category:Remote Server Access]]
[[Category:Department Servers]]

Running Jupyter Notebooks is now possible with Open OnDemand. To run a Notebook, follow the steps below. test

1 - Log in to '''Open OnDemand.'''

* https://ood.stat.cmu.edu/

2 - Click the '''Interactive Apps''' drop down menu and then select '''Jupyter Notebook.'''
[[File:OOD Jupyter Menu Bar.png|left|frameless|719x719px]]

3 - You will be taken to a new page where you can select options for the Notebook. Fill out the form as desired and then click the '''Launch''' button. Use [https://util.stat.cmu.edu/grafana/d/asjkldfbnvajdsfh213/leviathan-cluster?orgId=1&from=now-6h&to=now&timezone=browser Grafana] to help determine what server to use. Select one with low resource usage.

'''Hours''' - The number of hours your Notebook will run.

'''Server''' - The server to run your Notebook on. test

'''Jupyter Notebook Path''' - The path the Notebook environment lives on. You can use the default path which has many statistics related Python packages preinstalled, or you can define your own Notebook environment. See below for details on how to set your own environment up.

[[File:Jupyter Notebook Page.png|frameless|403x403px]]

4 - You will be taken to another new page where the Jupyter Notebook will load. Once it's ready, click '''Connect to Jupyter''' at the bottom. You will then be taken to a new tab with your Notebook session.

[[File:NotebookSession.png|frameless|526x526px]]

=== Custom Jupyter Notebook Environment ===
To build a custom Notebook environment, you can use a Python package manager such as Momba or Python's built in virtual environments. This example will cover Python virtual environments.

# Declare the virtual Python environment.
#* ''python3 -m venv /path/to/your/jupyter_env''
# Activate the virtual environment.
#*''source /path/to/your/jupyter_env/bin/activate''
# Install the Jupyter Notebook application to the new Python virtual environment.
#* ''pip install notebook''

To use your own Jupyter Environment, replace the default path above in step 3 for Jupyter Notebook Path.

Jupyter Notebooks

2026-08-11T19:10:27Z

Jacobg2:

[[Category:IDEs]]
[[Category:Remote Server Access]]
[[Category:Department Servers]]

Running Jupyter Notebooks is now possible with Open OnDemand. To run a Notebook, follow the steps below. test

1 - Log in to '''Open OnDemand.'''

* https://ood.stat.cmu.edu/

2 - Click the '''Interactive Apps''' drop down menu and then select '''Jupyter Notebook.'''
[[File:OOD Jupyter Menu Bar.png|left|frameless|719x719px]]

3 - You will be taken to a new page where you can select options for the Notebook. Fill out the form as desired and then click the '''Launch''' button. Use [https://util.stat.cmu.edu/grafana/d/asjkldfbnvajdsfh213/leviathan-cluster?orgId=1&from=now-6h&to=now&timezone=browser Grafana] to help determine what server to use. Select one with low resource usage.

'''Hours''' - The number of hours your Notebook will run.

'''Server''' - The server to run your Notebook on.

'''Jupyter Notebook Path''' - The path the Notebook environment lives on. You can use the default path which has many statistics related Python packages preinstalled, or you can define your own Notebook environment. See below for details on how to set your own environment up.

[[File:Jupyter Notebook Page.png|frameless|403x403px]]

4 - You will be taken to another new page where the Jupyter Notebook will load. Once it's ready, click '''Connect to Jupyter''' at the bottom. You will then be taken to a new tab with your Notebook session.

[[File:NotebookSession.png|frameless|526x526px]]

=== Custom Jupyter Notebook Environment ===
To build a custom Notebook environment, you can use a Python package manager such as Momba or Python's built in virtual environments. This example will cover Python virtual environments.

# Declare the virtual Python environment.
#* ''python3 -m venv /path/to/your/jupyter_env''
# Activate the virtual environment.
#*''source /path/to/your/jupyter_env/bin/activate''
# Install the Jupyter Notebook application to the new Python virtual environment.
#* ''pip install notebook''

To use your own Jupyter Environment, replace the default path above in step 3 for Jupyter Notebook Path.

Jupyter Notebooks

2026-08-11T18:56:28Z

Jacobg2:

Jupyter Notebooks

2026-08-11T18:56:05Z

Jacobg2:

[[Category:IDEs]]
[[Category:Remote Server Access]]
[[Category:Department Servers]]

Running Jupyter Notebooks is now possible with Open OnDemand. To run a Notebook, follow the steps below.

1 - Log in to '''Open OnDemand. test'''

* https://ood.stat.cmu.edu/

2 - Click the '''Interactive Apps''' drop down menu and then select '''Jupyter Notebook.'''
[[File:OOD Jupyter Menu Bar.png|left|frameless|719x719px]]

3 - You will be taken to a new page where you can select options for the Notebook. Fill out the form as desired and then click the '''Launch''' button. Use [https://util.stat.cmu.edu/grafana/d/asjkldfbnvajdsfh213/leviathan-cluster?orgId=1&from=now-6h&to=now&timezone=browser Grafana] to help determine what server to use. Select one with low resource usage.

'''Hours''' - The number of hours your Notebook will run.

'''Server''' - The server to run your Notebook on.

'''Jupyter Notebook Path''' - The path the Notebook environment lives on. You can use the default path which has many statistics related Python packages preinstalled, or you can define your own Notebook environment. See below for details on how to set your own environment up.

[[File:Jupyter Notebook Page.png|frameless|403x403px]]

4 - You will be taken to another new page where the Jupyter Notebook will load. Once it's ready, click '''Connect to Jupyter''' at the bottom. You will then be taken to a new tab with your Notebook session.

[[File:NotebookSession.png|frameless|526x526px]]

=== Custom Jupyter Notebook Environment ===
To build a custom Notebook environment, you can use a Python package manager such as Momba or Python's built in virtual environments. This example will cover Python virtual environments.

# Declare the virtual Python environment.
#* ''python3 -m venv /path/to/your/jupyter_env''
# Activate the virtual environment.
#*''source /path/to/your/jupyter_env/bin/activate''
# Install the Jupyter Notebook application to the new Python virtual environment.
#* ''pip install notebook''

To use your own Jupyter Environment, replace the default path above in step 3 for Jupyter Notebook Path.

Monitoring with Grafana

2026-08-11T18:55:36Z

Jacobg2:

[[Category:Wright Cluster]]
[[Category:Department Servers]]
[[Category:Remote Server Access]]

Grafana is a web based application that collects and displays server usage statistics within dedicated dashboards.

Each dashboard shows CPU usage, RAM usage, system load percentage, and currently logged in users.

[[File:Grafana Dashboard.png|thumb|alt=|border|none|479x479px]]

Use these dashboards to help decide which server to use for your work. Choose a server that has a low resource usage and few logged in users.

Click the links below to be taken to each dashboard.

[https://util.stat.cmu.edu/grafana/d/asjkldfbnvajdsfh213/leviathan-cluster?orgId=1&from=now-6h&to=now&timezone=browser Leviathans]

[https://util.stat.cmu.edu/grafana/d/beuw1q6y3dgjkc/hydra-cluster?orgId=1&from=now-6h&to=now&timezone=browser Hydras]

[https://util.stat.cmu.edu/grafana/d/edkar1rgy3l6od/ghidorah-cluster?orgId=1&from=now-30m&to=now&timezone=browser Ghidorahs]

Monitoring with Grafana

2026-08-11T18:55:29Z

Jacobg2:

[[Category:Wright Cluster]]
[[Category:Department Servers]]
[[Category:Remote Server Access]]

Grafana is a web based application that collects and displays server usage statistics within dedicated dashboards.

Each dashboard shows CPU usage, RAM usage, system load percentage, and currently logged in users.

[[File:Grafana Dashboard.png|thumb|alt=|border|none|479x479px]]

Use these dashboards to help decide which server to use for your work. Choose a server that has a low resource usage and few logged in users.

Click the links below to be taken to each dashboard. test

[https://util.stat.cmu.edu/grafana/d/asjkldfbnvajdsfh213/leviathan-cluster?orgId=1&from=now-6h&to=now&timezone=browser Leviathans]

[https://util.stat.cmu.edu/grafana/d/beuw1q6y3dgjkc/hydra-cluster?orgId=1&from=now-6h&to=now&timezone=browser Hydras]

[https://util.stat.cmu.edu/grafana/d/edkar1rgy3l6od/ghidorah-cluster?orgId=1&from=now-30m&to=now&timezone=browser Ghidorahs]

Monitoring with Grafana

2026-08-11T18:41:23Z

Jacobg2:

Monitoring with Grafana

2026-08-11T18:40:40Z

Jacobg2:

[[Category:Wright Cluster]]
[[Category:Department Servers]]
[[Category:Remote Server Access]]

Grafana is a web based application that collects and displays server usage statistics within dedicated dashboards.

Each dashboard shows CPU usage, RAM usage, system load percentage, and currently logged in users.

[[File:Grafana Dashboard.png|thumb|alt=|border|none|479x479px]]

Use these dashboards to help decide which server to use for your work. Choose a server that has a low resource usage and few logged in users.

Click the links below to be taken to each dashboard. test

[https://util.stat.cmu.edu/grafana/d/asjkldfbnvajdsfh213/leviathan-cluster?orgId=1&from=now-6h&to=now&timezone=browser Leviathans]

[https://util.stat.cmu.edu/grafana/d/beuw1q6y3dgjkc/hydra-cluster?orgId=1&from=now-6h&to=now&timezone=browser Hydras]

[https://util.stat.cmu.edu/grafana/d/edkar1rgy3l6od/ghidorah-cluster?orgId=1&from=now-30m&to=now&timezone=browser Ghidorahs]

Main Page

2026-08-11T18:31:00Z

Jacobg2:

Main Page

2026-08-11T18:30:29Z

Jacobg2:

Directory of Computing Resources for Statistical Research

2026-08-10T13:17:24Z

Jacobg2: /* Other Resources */

'''Computing Resources''' are essential for modern statistical research and data science. The department, university, and national organizations provide several High-Performance Computing (HPC) clusters to scale up your workflows, run massive simulations, or train deep learning models.

This page provides an overview of the computing clusters available to members of the department, detailing who manages them, hardware capabilities, and where to find setup instructions.

<div style="background-color: #e2eef9; border-left: 5px solid #005A9C; padding: 15px; margin-bottom: 20px; margin-top: 15px;">
'''Tip: Choosing the Right Cluster'''<br>
Start local before scaling up. Use the department-managed clusters for prototyping, data preparation, and medium-scale CPU jobs. For massive parallelism or extensive GPU needs that exceed department limits, you should apply for time on external clusters like PSC Bridges-2.
</div>

== Available Computing Clusters ==
'''Note: We are going to be decommissioning the old hydra servers. Please move your workflows to the leviathan servers!'''
{| class="wikitable" style="width: 100%; border-collapse: collapse; text-align: left; box-shadow: 0 2px 5px rgba(0,0,0,0.1);"
|- style="background-color: #f8f9fa; border-bottom: 2px solid #dee2e6;"
! style="padding: 12px; width: 10%;" | HPC Cluster Name
! style="padding: 12px; width: 15%;" | Management
! style="padding: 12px; width: 10%;" | Has GPUs?
! style="padding: 12px; width: 8%;" | Sharing Policy
! style="padding: 12px; width: 20%;" | How to access
! style="padding: 12px; width: 37%;" | Setup & Documentation Articles
|- style="border-bottom: 1px solid #dee2e6;"
| style="padding: 12px; font-weight: bold;" | Leviathans
| style="padding: 12px;" | Statistics Department
| style="padding: 12px;" | No
| style="padding: 12px;" | FCFS
| style="padding: 12px;" | Anytime direct ssh to server, e.g. <code>ssh [userid]@leviathan1.stat.cmu.edu</code>
| style="padding: 12px;" |
<categorytree mode="pages" depth="1" hideroot="on">Department Servers</categorytree>
|- style="border-bottom: 1px solid #dee2e6;"
| style="padding: 12px; font-weight: bold;" | Ghidorahs
| style="padding: 12px;" | Statistics Department
| style="padding: 12px; color: #28a745; font-weight: bold;" | Yes
(3 nodes)
| style="padding: 12px;" | FCFS
| style="padding: 12px;" | Anytime direct ssh to server, e.g. <code>ssh [userid]@ghidorah1.stat.cmu.edu</code>
| style="padding: 12px;" |
<categorytree mode="pages" depth="1" hideroot="on">Department Servers</categorytree>
|- style="border-bottom: 1px solid #dee2e6; background-color: #fdfdfd;"
| style="padding: 12px; font-weight: bold;" | Wright Cluster
| style="padding: 12px;" | Collaboration between several CMU departments
| style="padding: 12px; color: #28a745; font-weight: bold;" | Yes
(3 nodes)
| style="padding: 12px;" | Slurm
| style="padding: 12px;" | See instructions [[Wright GPU Cluster|here]]
| style="padding: 12px;" |
<categorytree mode="pages" depth="1" hideroot="on">Wright Cluster</categorytree>
|- style="border-bottom: 1px solid #dee2e6; background-color: #fdfdfd;"
| style="padding: 12px; font-weight: bold;" | PSC Clusters
| style="padding: 12px;" | External (Pittsburgh Supercomputing Center)
| style="padding: 12px; color: #28a745; font-weight: bold;" | Yes
(Many nodes)
| style="padding: 12px;" | Slurm
| style="padding: 12px;" | See instructions [[PSC|here]]
| style="padding: 12px;" |
<categorytree mode="pages" depth="1" hideroot="on">PSC</categorytree>
|}

<div style="background-color: #fff3cd; border-left: 5px solid #ffc107; padding: 15px; margin-top: 20px;">
'''Note on Accounts:''' Department-managed clusters usually require you to request access via the department IT staff or faculty sponsor. External resources like PSC require writing a brief research proposal and creating an ACCESS account.
</div>

== Technical Specifications ==

The following table summarizes the hardware specifications per node for each resource type:

{| class="wikitable" style="width: 100%; text-align: left;"
|-
! Cluster
! Number of Nodes
! CPUs per Node
! RAM per Node
! GPUs per Node
! Notes
|-
| '''Leviathans'''
| 11
| 16 cores (AMD Ryzen 9 9950x @ 5.7GHz)
| 128GB (Nodes 1-6)<br>196GB (Nodes 7-11)
| 0
|
|-
| '''Ghidorahs'''
| 3
| 24 cores (Intel Core i9-13900KF: 8 P-cores, 16 E-cores)
| 64GB
| 1 (NVIDIA RTX 4090 24GB)
|
|-
| '''Wright Cluster'''
| 3
| 64 cores
| 1TB
| 8 (NVIDIA L40 48GB)
| See [https://www.cmu.edu/dietrich/statistics-datascience/resources/docs/wright-gpu-cluster-policy.pdf Wright GPU Cluster Policy - CMU]
|-
| '''PSC Clusters'''
| >500
| Varies
| Varies
| Varies
| See [https://www.psc.edu/resources/bridges-2/ Bridges-2 Nodes]
|}

== Software ==
The servers within Statistics & Data Science offer a variety of software. Sudo access, also referred to as the ability to install software, on department servers is not allowed. If a software package you would like to use is not listed below, please contact remarks@stat.cmu.edu.
{| class="wikitable"
|+
!
!Ghidorahs
!Leviathans
|-
|Python
|X
|X
|-
|R
|X
|X
|-
|Apptainer
|X
|X
|-
|Quarto
|
|X
|-
|Gurobi
|
|X
|-
|Rstudio-Server
|
|X
|-
|Matlab
|
|X
|-
|Julia
|
|X
|}

== Other Resources ==
* [[Open OnDemand]]
* [[Open OnDemand Wright]]
* [[Monitoring with Grafana]]
* [https://util.stat.cmu.edu/grafana/dashboards/f/efud3z15idnuoa/ Grafana Monitor]

Jupyter Notebooks

2026-08-06T13:05:54Z

Jacobg2:

Linux and department servers

2026-08-06T13:04:56Z

Jacobg2: /* Checking current usage */

[[Category:Department Servers]]
[[Category:Remote Server Access]]

The department has two sets of servers, the Leviathans and the Hydras, that can help you run large analyses and gain a better understanding of common command line structure for more high scale computing. One can also view the department servers a bit more as a “sandbox” or “playground” as there are very few constraints and it’s okay to not run as polished code as you might on the [[PSC]] or [[Wright GPU Cluster]].

['''Request:''' All users - if you see comments that are outdated / less helpful than expected please contribute.]

== What resources exist ==

# The department provides 11 leviathan servers (named {<code>leviathan1</code>, ..., <code>leviathan11</code>}<code>.stat.cmu.edu</code>).
# The department provides 3 Ghidorah GPU servers (named {<code>ghidorah1</code>, ..., <code>ghidorah3</code>}<code>.stat.cmu.edu</code>).
# The Ghidorahs and Leviathans all share the same data file system. This allows you to upload a file to any of the servers and then have it on all of them.

''See [[what_resources_when| What resources when]] for guides to help decide which resource you should be using.''

==Introduction to connecting to the server and useful Linux commands==

The department servers all run Linux and require you to interact with them remotely. While you can use a traditional command line, modern IDEs like VS Code also allow for highly integrated graphical interaction.

===Getting Access: (<code>ssh</code>)===

To get onto the server and execute code, you’ll need to access it through <code>ssh</code>. Modern operating systems (including Mac, Linux, and Windows 10/11) have this tool built-in natively.

<ul>
<li>'''Mac / Linux:''' Open up a <code>Terminal</code>.</li>
<li>'''Windows:''' Open up <code>Windows PowerShell</code> or <code>Windows Terminal</code>.</li>
</ul>

In your terminal, type:
<syntaxhighlight lang="bash">ssh andrew_id@leviathan1.stat.cmu.edu</syntaxhighlight>

You will then be prompted to type in your password. (Note: as you type, no characters will appear on the screen for security reasons. Just type it and hit Enter). If you don’t know your password or have never accessed the server before, email [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu].

If you’re accessing the servers frequently, we highly recommend setting up passwordless SSH (see below).

=== Passwordless SSH ===
To connect to the server without typing in your password every time, you need to generate an SSH key pair. This creates a private key that stays securely on your laptop, and a public key that the servers use to recognize you.

* Open up your terminal (Mac/Linux) or PowerShell (Windows).
* Check whether you already have a key pair by looking in your <code>.ssh</code> folder (run <code>ls ~/.ssh</code> on Mac/Linux, or <code>dir ~\.ssh</code> on Windows). If you have files like <code>id_ed25519</code> or <code>id_rsa</code>, you already have a key pair and can skip to the upload step.
* To generate a new key pair, we recommend the Ed25519 algorithm. Run:
<syntaxhighlight lang="bash">ssh-keygen -t ed25519</syntaxhighlight>
* Hit Enter to accept the default file location. When prompted for a passphrase, you can enter one for extra security or leave it empty for true passwordless login.
* Next, upload the public key to the remote server. Because the hydras and ghidorahs share a file system, you only need to do this once.
** '''For Mac/Linux:''' Use the built-in <code>ssh-copy-id</code> command:
** <syntaxhighlight lang="bash">ssh-copy-id andrew_id@leviathan1.stat.cmu.edu</syntaxhighlight>
** '''For Windows:''' Windows OpenSSH does not natively include the <code>ssh-copy-id</code> command. Instead, use this PowerShell one-liner to securely copy your key over:
** <syntaxhighlight lang="powershell">Get-Content "$env:USERPROFILE\.ssh\id_ed25519.pub" | ssh andrew_id@leviathan1.stat.cmu.edu "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys"</syntaxhighlight>
* To ensure the permissions of the <code>~/.ssh</code> directory on the server are strictly enforced, log into the server normally and run:
<syntaxhighlight lang="bash">
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
</syntaxhighlight>
You should now be able to SSH into the server without being prompted for a password.

===File sharing (<code>scp</code> + GUIs)===

There are two primary ways to move files between your personal computer and the server: using the command line (<code>scp</code>) or using a graphical interface (like VS Code or Cyberduck).

====Command line <code>scp</code>====

Because Windows now includes OpenSSH natively, the <code>scp</code> (secure copy) command works exactly the same across Mac, Linux, and Windows PowerShell.

<ol>
<li>Copy from server to your computer
<syntaxhighlight lang="bash">scp andrew_id@leviathan1.stat.cmu.edu:/path/to/remote/file.txt /local/directory/</syntaxhighlight>
To copy an entire folder, add the <code>-r</code> (recursive) flag:
<syntaxhighlight lang="bash">scp -r andrew_id@leviathan1.stat.cmu.edu:/path/to/remote/folder/ /local/directory/</syntaxhighlight>
</li>
<li>Copy from your computer to the server
<syntaxhighlight lang="bash">scp /local/file.txt andrew_id@leviathan1.stat.cmu.edu:/remote/directory/</syntaxhighlight>
</li>
</ol>

====GUI “drag-and-drop” file sharing====
Many students prefer graphical user interfaces (GUIs) to transfer files instead of using the command line. These tools allow you to visually drag and drop files from your computer to the server (and vice-versa).

* '''Visual Studio Code (Recommended for all OS):''' If you use VS Code with the Remote-SSH extension (see "Text Editors" below), you can simply drag and drop files from your desktop directly into the VS Code file explorer pane. This is the easiest and most integrated method.
* '''Mac:''' [https://cyberduck.io/ Cyberduck] remains a solid dedicated tool.
* '''Windows:''' [https://winscp.net/eng/download.php WinSCP] is a popular dedicated GUI, though legacy SSH clients like PuTTY are no longer necessary for modern Windows file transfers.

=== Basic CLI (Linux command line): <code>bash</code>===

Understanding the basic Linux command line is important to navigate when you are on the server. Below are some basic commands to move around the file system and manipulate files.

{| class="wikitable"
|-
!command
!translation
|-
|
<code>pwd</code>
|print the current working directory
|-
|
<code>cd _dir_</code>
| change directory
|-
|
<code>rm _file_</code>
|remove / delete file
|-
|
<code>mv _file_ _dir_</code>
|move file (can change name)
|-
|
<code>cp _file_ _dir_</code>
|copy file
|-
|
<code>ls _dir_</code>
|list files in directory
|}

Use <code>man _command_</code> to get a listing / manual of the command.

Additional resources geared toward statistical computing can be found here: [http://www.jarrodmillman.com/rcsds/standard/bash.html UCB 2015 reproducible DS lecture notes].

'''TODO: add information about <code>sed</code>, <code>chmod</code>, and a link to bash files.'''

==Text editors and IDEs==

Historically, editing code on a server required using terminal-based text editors. While still useful for quick changes, modern tools allow you to use a full graphical IDE on your laptop that computes natively on the server.

=== Visual Studio Code (Highly Recommended) ===
[https://code.visualstudio.com/ Visual Studio Code] (VS Code) is the industry standard for remote development. By using the Microsoft '''Remote - SSH''' extension, you can connect VS Code directly to the department servers.

# Install VS Code on your personal machine.
# Go to the Extensions tab and install '''Remote - SSH'''.
# Click the remote indicator (the green arrows) in the bottom-left corner of VS Code, select "Connect to Host...", and type <code>andrew_id@leviathan1.stat.cmu.edu</code>.
# Once connected, you can open server folders, edit scripts with full syntax highlighting, run interactive terminal sessions, and drag-and-drop files between your laptop and the server all from within the VS Code window.

=== Terminal Editors ===
If you only need to make a quick tweak to a file while already logged into the terminal, you can use built-in editors:
* <code>nano _filename_</code>: A very simple, beginner-friendly text editor. Use <code>Ctrl+O</code> to save and <code>Ctrl+X</code> to exit.
* <code>vim _filename_</code>: A highly powerful but steep-learning-curve text editor.

==Using the servers respectfully==
Because the servers are a shared department resource it is important to use them respectfully. There are many servers and multiple cores (CPUs) per server. To be respectful of other users:

#'''Check the current usage''' of the server you're on before using it.
#Do not use '''all''' the cores on a server, and do not use '''all''' the servers at the same time, so others can get work done.
#If you have a long-running job, '''deprioritize it''' so others can run too.
#Use <code>/scratch</code> for jobs that require intense disk I/O operations

Each of these is easy to check.

=== Checking current usage===
To check which servers are currently being used and which are open, see the [https://util.stat.cmu.edu/grafana/d/beuw1q6y3dgjkc/hydra-cluster?orgId=1&from=now-6h&to=now&timezone=browser Grafana dashboard for the hydras] and [https://util.stat.cmu.edu/grafana/d/asjkldfbnvajdsfh213/leviathan-cluster?orgId=1&from=now-6h&to=now&timezone=browser for the Leviathans]. It lists them all and their current usage.

When you're logged in to one node, you can use the <code>top</code> command. <code>top</code> displays the current usage of the server’s resources with a list of jobs run. It also tells you the percentage of the cores that are being used and the amount of RAM (memory) that is available. See [https://www.howtogeek.com/668986/how-to-use-the-linux-top-command-and-understand-its-output/ howtogeek] resources on how to interpret it all.

===Limiting the cores you use===
We recommend that when using parallel processing, use fewer than the total number of cores on the server. Most parallel processing packages let you choose the number of cores to use; the hydras have 12, so pick something smaller.

=== Deprioritize when possible===
<code>nice/renice</code> allows you to lower the priority of your jobs, so the operating system lets other processes run first. See [https://www.howtogeek.com/411979/how-to-set-process-priorities-with-the-nice-and-renice-commands-in-linux/ howtogeek] resources on the basics. <code>nice</code> sets the priority, and <code>renice</code> updates your priority. On the department servers, everyone starts at the same priority, and you can only '''deprioritize''' your jobs -- you can't make them higher priority than everyone else.<syntaxhighlight lang="bash">nice -20 Rscript some-big-script.R
renice -n 20 processid # get the process ID from top</syntaxhighlight>

===Using <code>/scratch</code>===
On department servers, your home directory (the default location you start in when you ssh) is not actually located on the physical server itself. Instead, every department server uses network attached storage (NAS), where everybody's home directly lives on a central storage server and the individual Leviathans and Ghidorahs connect to that storage server. This is the functionality that lets your files follow you no matter what node you log into.

While computing processes on one server do not affect processes on other servers, every server shares the same NAS. Therefore, '''if a user is reading and writing a ton of data to their home directory, it will slow down reading and writing for everyone, even if you aren't on the same server as this user.''' Even if your job does not read or write much data, disk I/O slowdowns will affect common operations, such as ssh, importing packages, and uploading/downloading files.

If you need to run a job that requires a ton of reading and writing data to disk, '''do not read/write using your home directory'''. Instead, use the <code>/scratch</code> folder located on every server. This folder points to an SSD that is specific to that server. Not only will read and writes to <code>/scratch</code> not affect users on other servers, but it will also be generally faster than using your home directory even if nobody else is online. '''If you plan to use''' <code>/scratch</code>''', you should make a new folder inside of''' <code>/scratch</code> '''first and put your data inside that folder. For example:'''<syntaxhighlight lang="bash">
cd /scratch
mkdir my_data # Default permission allows only you to modify contents
cd my_data
# Do stuff in my_data
</syntaxhighlight>It is also very important to remember that <code>/scratch</code> '''is specific to each Leviathan and Ghidorah and will not follow you between servers'''. Thus, you should not use <code>/scratch</code> for long-term storage or for important data. In addition, space on <code>/scratch</code> is limited, so be sure to clean up your data once you don't need it anymore.
==Running jobs==

We have included thoughts about running jobs in two places.

# The first looks at tools that allow you to keep running jobs on the server when you close the connection to the server (turn off your laptop, etc.). These tools are call [[terminal multiplexers]]. 
#The second set of tools focus on how to run your [[R_on_the_server|R analysis]] or [[Python_on_the_server|Python analysis]] code on the server.

==Server specs==

Generally speaking, our servers have multiple CPU cores, but no GPU cores. For deep learning tasks that need GPUs, consider using the [[GPU Cluster|GPU cluster]].

To examine the specs yourself, use
<syntaxhighlight lang="bash">cat /proc/meminfo
cat /proc/cpuinfo</syntaxhighlight>

to see some information.

Department Servers

2026-08-06T12:57:35Z

Jacobg2: /* Ghidorahs */

[[Category:Delete Me]]

Delete Me: Integrated into Directory of Resources

== Using the Servers ==

Accessing the servers requires using an SSH compatible terminal. Applications such as "Command Prompt" on Windows or "Terminal" on MacOS can achieve this.
To SSH to a server, open either app listed previously and enter the SSH command shown below. Be sure to use your StatDS computing account.<syntaxhighlight lang="bash">
ssh statdsusername@leviathan11.stat.cmu.edu
</syntaxhighlight>In the line above, replace "hydra1" with the server you want to connect to.
{| class="wikitable"
|+
Servers
!Hydras
!Ghidorahs
!Leviathans
|-
|hydra1
|ghidorah1
|leviathan1
|-
|hydra2
|ghidorah2
|leviathan2
|-
|hydra3
|ghidorah3
|leviathan3
|-
|hydra4
|
|leviathan4
|-
|
|
|leviathan5
|-
|
|
|leviathan6
|-
|
|
|leviathan7
|-
|
|
|leviathan8
|-
|
|
|leviathan9
|-
|
|
|leviathan10
|-
|
|
|leviathan11
|}

== Hydras ==
The Hydras are virtual based serves hosted in the Campus Cloud. They offer slightly higher stability and are better for smaller computing jobs.

Hardware Specifications

* Intel Xeon CPU
** 12 cores @2.6Ghz
* 64GB of RAM
* NAS used for /home storage
Servers = Hydra1-4

ssh yourusername@hydra1.stat.cmu.edu

[https://util.stat.cmu.edu/grafana/d/beuw1q6y3dgjkc/hydra-cluster?orgId=1&from=now-6h&to=now&timezone=browser Grafana Monitoring]

== Ghidorahs ==
The Ghidorahs are physical servers located throughout Baker Hall. They are designed for lighter GPU jobs.

Hardware Specifications

* Core i9-13900KF CPU
** 8 performance cores @5.4Ghz
** 16 Efficiency cores @4.3Ghz
* 64GB of RAM
* RTX 4090 24GB of video memory
* NAS used for /home storage

[https://util.stat.cmu.edu/grafana/d/edkar1rgy3l6od/ghidorah-cluster?orgId=1&from=now-30m&to=now&timezone=browser Grafana Monitoring]
== Leviathans ==
The Leviathans are physical servers located throughout Baker Hall and the Cyert Data Center. They are designed for heavy CPU and RAM jobs.

Hardware Specifications

* AMD Ryzen 9 9950x
** 16 cores @5.7Ghz
* 128GB of RAM (Leviathan1-6)
* 196GB of RAM (Leviathan7-11)
* 4TB of local NVME storage
** Mounted at /scratch
** Data stored in this location will only be accessible on the original server.
* NAS used for /home storage
Servers = Leviathan1-11

ssh yourusername@leviathan1.stat.cmu.edu

* You can use "levi" in place of "leviathan"
** ''ssh yourusername@levi1.stat.cmu.edu''

[https://util.stat.cmu.edu/grafana/d/asjkldfbnvajdsfh213/leviathan-cluster?orgId=1&from=now-6h&to=now&timezone=browser Grafana Monitor]

== Storage ==
Each server's "/home" directory located on the department's NAS. This means your files will be accessible on every server.

The NAS is a total of 100TB. This space is shared and monitored for fair usage. If you store large amounts of data, you may be asked if it is all necessary.
To see available space on the NAS, run the command below in a terminal on the server.<syntaxhighlight lang="bash">
df -h | grep "/home"
</syntaxhighlight>To see your total usage run the command below. ''It may take a few minutes to finish''<syntaxhighlight lang="bash">
du -sh ~
</syntaxhighlight>

[https://util.stat.cmu.edu/grafana/d/i55CRAlMk/synology-nas-details?orgId=1&from=now-3h&to=now&timezone=browser&var-interfacebits=8&var-JobName=crater&var-Device=crater.stat.cmu.edu:9116&var-InterfacePhysical=$__all&var-InterfaceVirtual=$__all&var-DiskName=$__all&var-DiskNameSSD=$__all&var-raidName=$__all&var-storageIODevice=$__all&var-spaceIODevice=$__all&var-hrStorageDescr=$__all&var-flashCacheDisk=$__all&var-hrMemoryDescr=$__all&var-SmartDev=$__all&var-services=$__all&refresh=5m Grafana Monitor]
== Software ==
The servers within Statistics & Data Science offer a variety of software. Sudo access, also referred to as the ability to install software, on department servers is not allowed. If a software package you would like to use is not listed below, please contact remarks@stat.cmu.edu.
{| class="wikitable"
|+
!
!Hydras
!Ghidorahs
!Leviathans
|-
|Python
|X
|X
|X
|-
|R
|X
|X
|X
|-
|Apptainer
|X
|X
|X
|-
|Quarto
|X
|
|X
|-
|Gurobi
|X
|x
|X
|-
|Rstudio-Server
|X
|
|X
|-
|Matlab
|X
|
|X
|-
|Julia
|X
|
|X
|}

Department Servers

2026-08-06T12:56:14Z

Jacobg2: /* Hydras */

[[Category:Delete Me]]

Delete Me: Integrated into Directory of Resources

== Using the Servers ==

Accessing the servers requires using an SSH compatible terminal. Applications such as "Command Prompt" on Windows or "Terminal" on MacOS can achieve this.
To SSH to a server, open either app listed previously and enter the SSH command shown below. Be sure to use your StatDS computing account.<syntaxhighlight lang="bash">
ssh statdsusername@leviathan11.stat.cmu.edu
</syntaxhighlight>In the line above, replace "hydra1" with the server you want to connect to.
{| class="wikitable"
|+
Servers
!Hydras
!Ghidorahs
!Leviathans
|-
|hydra1
|ghidorah1
|leviathan1
|-
|hydra2
|ghidorah2
|leviathan2
|-
|hydra3
|ghidorah3
|leviathan3
|-
|hydra4
|
|leviathan4
|-
|
|
|leviathan5
|-
|
|
|leviathan6
|-
|
|
|leviathan7
|-
|
|
|leviathan8
|-
|
|
|leviathan9
|-
|
|
|leviathan10
|-
|
|
|leviathan11
|}

== Hydras ==
The Hydras are virtual based serves hosted in the Campus Cloud. They offer slightly higher stability and are better for smaller computing jobs.

Hardware Specifications

* Intel Xeon CPU
** 12 cores @2.6Ghz
* 64GB of RAM
* NAS used for /home storage
Servers = Hydra1-4

ssh yourusername@hydra1.stat.cmu.edu

[https://util.stat.cmu.edu/grafana/d/beuw1q6y3dgjkc/hydra-cluster?orgId=1&from=now-6h&to=now&timezone=browser Grafana Monitoring]

== Ghidorahs ==
The Ghidorahs are physical servers located throughout Baker Hall. They are designed for lighter GPU jobs.

Hardware Specifications

* Core i9-13900KF CPU
** 8 performance cores @5.4Ghz
** 16 Efficiency cores @4.3Ghz
* 64GB of RAM
* RTX 4090 24GB of video memory
* NAS used for /home storage

[https://info.stat.cmu.edu/grafana/d/edkar1rgy3l6od/ghidorah-cluster?orgId=1 Grafana Monitoring]
== Leviathans ==
The Leviathans are physical servers located throughout Baker Hall and the Cyert Data Center. They are designed for heavy CPU and RAM jobs.

Hardware Specifications

* AMD Ryzen 9 9950x
** 16 cores @5.7Ghz
* 128GB of RAM (Leviathan1-6)
* 196GB of RAM (Leviathan7-11)
* 4TB of local NVME storage
** Mounted at /scratch
** Data stored in this location will only be accessible on the original server.
* NAS used for /home storage
Servers = Leviathan1-11

ssh yourusername@leviathan1.stat.cmu.edu

* You can use "levi" in place of "leviathan"
** ''ssh yourusername@levi1.stat.cmu.edu''

[https://info.stat.cmu.edu/grafana/d/asjkldfbnvajdsfh213/leviathan-cluster?orgId=1 Grafana Monitor]

== Storage ==
Each server's "/home" directory located on the department's NAS. This means your files will be accessible on every server.

The NAS is a total of 100TB. This space is shared and monitored for fair usage. If you store large amounts of data, you may be asked if it is all necessary.
To see available space on the NAS, run the command below in a terminal on the server.<syntaxhighlight lang="bash">
df -h | grep "/home"
</syntaxhighlight>To see your total usage run the command below. ''It may take a few minutes to finish''<syntaxhighlight lang="bash">
du -sh ~
</syntaxhighlight>

== Software ==
The servers within Statistics & Data Science offer a variety of software. Sudo access, also referred to as the ability to install software, on department servers is not allowed. If a software package you would like to use is not listed below, please contact remarks@stat.cmu.edu.
{| class="wikitable"
|+
!
!Hydras
!Ghidorahs
!Leviathans
|-
|Python
|X
|X
|X
|-
|R
|X
|X
|X
|-
|Apptainer
|X
|X
|X
|-
|Quarto
|X
|
|X
|-
|Gurobi
|X
|
|X
|-
|Rstudio-Server
|X
|
|X
|-
|Matlab
|X
|
|X
|-
|Julia
|X
|
|X
|}

Monitoring with Grafana

2026-08-06T12:55:00Z

Jacobg2:

Open OnDemand

2026-07-07T14:20:53Z

Jacobg2: /* Getting Started */

[[Category:Wright Cluster]]
[[Category:Department Servers]]
[[Category:Remote Server Access]]

Open OnDemand provides easy, remote web access to supercomputers and computing clusters, without the need for a command-line interface. Instead of accessing the hydra and ghidorah clusters via your terminal, Open OnDemand provides a user interface for accessing and modifying files and submitting jobs. No installation is required -- everything can be accessed within the browser.

=== Getting Started ===
To access Open OnDemand, login [https://ood.stat.cmu.edu/ here] with your StatDS username and password. This is the same username and password you would use to access the clusters via ssh on your terminal.

After logging in, you should be redirected to your file system. You can navigate through your directories just as you would on your desktop. A nice feature of Open OnDemand is you can automatically upload and download files from your desktop to the department's file system; the secure file transfer is done automatically in a user-friendly manner.

=== Editing Files ===
To create a new file or directory, simply click "New File" or "New Directory" in the upper right hand corner of the screen.

To edit existing files, navigate to the file in the file system. Click on the dropdown three vertical dots to the right of the file's name. An options list will appear that includes a "View" mode and an "Edit" mode. Clicking view will allow you to view the contents of the file without manipulating them. Edit will redirect you to another tab where you can directly edit the contents of the file. Click the "Save" button on the lefthand side of the screen to save edits; when this button is grayed out, it means there are no additional changes to save.

=== Accessing the Clusters ===
To access the head node via the terminal, navigate to the "Clusters" drop down in the top left of the screen, and select "Statistics and Data Science Shell Access." The shell allows you to access your files within a terminal on the head compute node. Do not run jobs within this terminal. Beneath the cluster selection, you can view the system status which tells you how many nodes, processors, and GPUs are currently available, as well as how many jobs are running and queued. This will help inform how long a job you've created might take to run on the system/when it will begin running.

=== Submitting and Tracking Jobs ===

==== Default Template ====
To submit or compose a job, navigate to the "Jobs" drop down in the top left of the screen, and select "Job Composer." This will take you to the job composer page, where you can select the "New Job" drop down. To start from the default template, just click "From Default Template" and in the right hand panel, scroll down to where the .sh script is shone and click "Open Editor" to edit the script itself. Once you are ready to submit the job, simply click the job in your list of potential jobs, and click submit. The job will show a status of "Queued," "Running," or "Completed" and you can view output files in the right hand panel once it has completed running.

==== Hydra, Leviathan, and Ghidorah Templates ====
If you click "From Template," you can choose to submit a job on the hydras, leviathans, or ghidorahs. Simply pick the template you want to use -- either Hydra Job for the hydras or Ghidorah Job for the ghidorahs -- and click "Create New Job." You can then modify the submit.sh script as before to select ''which'' server you want to use as follows:<syntaxhighlight lang="bash">
#!/bin/sh
#SBATCH --partition Ghidorahs

# #Select a specific Hydra node to run job on.
# #Un-comment the line below and replace ghidorahXX with the appropriate ghidorah server ~ ghidorah1-3.
# #SBATCH --nodelist=ghidorahx

# #Output file for script.
#SBATCH --output=%j_output.txt

# #Error file for script.
#SBATCH --error=%j_error.txt

python3 examplepython.py
</syntaxhighlight>The <code>#SBATCH --nodelist=ghidorah1</code> selects the 1st ghidorah to run the python script on. Change this to <code>#SBATCH --nodelist=hydra1</code> for the hydras and change the first SBATCH line to Hydras (<code>#SBATCH --partition Hydras</code>).

==== Submitting a Script from a Specified Path ====
You can also specify a path to a job you have already created. Click "From Specified Path" and enter the source path, i.e. the path to the directory containing your .sh file. Once you have done that, you can specify the name of the script under Script name, i.e. the name of the .sh file in the directory you've navigated to. You can then click "Run" just as you have with your other jobs.

==== Limiting Resource Usage ====
Limiting your resource usage on servers can help prevent server locks. Use a submit script, similar to the one below, to limit the number of CPU cores and RAM.<syntaxhighlight lang="bash">
#!/bin/sh
#SBATCH --partition Leviathans
#SBATCH --nodelist=leviathan3

# #Output file for script.
#SBATCH --output=%j_output.txt

# #Error file for script.
#SBATCH --error=%j_error.txt

# # Requests number of CPUs per task
#SBATCH --ntasks=1
#SBATCH --cpus-per-task 30

# # Request amount of memory (RAM)
#SBATCH --mem=110G

python3 examplepython.py
</syntaxhighlight>

==== Wright GPU Cluster ====
Open OnDemand is also available for the Wright GPU cluster. Documentation can be found [[Open OnDemand Wright|here]].

Department Servers

2026-06-25T12:13:59Z

Jacobg2:

== Using the Servers ==

Accessing the servers requires using an SSH compatible terminal. Applications such as "Command Prompt" on Windows or "Terminal" on MacOS can achieve this.
To SSH to a server, open either app listed previously and enter the SSH command shown below. Be sure to use your StatDS computing account.<syntaxhighlight lang="bash">
ssh statdsusername@leviathan11.stat.cmu.edu
</syntaxhighlight>In the line above, replace "hydra1" with the server you want to connect to.
{| class="wikitable"
|+
Servers
!Hydras
!Ghidorahs
!Leviathans
|-
|hydra1
|ghidorah1
|leviathan1
|-
|hydra2
|ghidorah2
|leviathan2
|-
|hydra3
|ghidorah3
|leviathan3
|-
|hydra4
|
|leviathan4
|-
|
|
|leviathan5
|-
|
|
|leviathan6
|-
|
|
|leviathan7
|-
|
|
|leviathan8
|-
|
|
|leviathan9
|-
|
|
|leviathan10
|-
|
|
|leviathan11
|}

== Hydras ==
The Hydras are virtual based serves hosted in the Campus Cloud. They offer slightly higher stability and are better for smaller computing jobs.

Hardware Specifications

* Intel Xeon CPU
** 12 cores @2.6Ghz
* 64GB of RAM
* NAS used for /home storage
Servers = Hydra1-4

ssh yourusername@hydra1.stat.cmu.edu

[https://info.stat.cmu.edu/grafana/d/beuw1q6y3dgjkc/hydra-cluster?orgId=1 Grafana Monitoring]
== Ghidorahs ==
The Ghidorahs are physical servers located throughout Baker Hall. They are designed for lighter GPU jobs.

Hardware Specifications

* Core i9-13900KF CPU
** 8 performance cores @5.4Ghz
** 16 Efficiency cores @4.3Ghz
* 64GB of RAM
* RTX 4090 24GB of video memory
* NAS used for /home storage

[https://info.stat.cmu.edu/grafana/d/edkar1rgy3l6od/ghidorah-cluster?orgId=1 Grafana Monitoring]
== Leviathans ==
The Leviathans are physical servers located throughout Baker Hall and the Cyert Data Center. They are designed for heavy CPU and RAM jobs.

Hardware Specifications

* AMD Ryzen 9 9950x
** 16 cores @5.7Ghz
* 128GB of RAM (Leviathan1-6)
* 196GB of RAM (Leviathan7-11)
* 4TB of local NVME storage
** Mounted at /scratch
** Data stored in this location will only be accessible on the original server.
* NAS used for /home storage
Servers = Leviathan1-11

ssh yourusername@leviathan1.stat.cmu.edu

* You can use "levi" in place of "leviathan"
** ''ssh yourusername@levi1.stat.cmu.edu''

[https://info.stat.cmu.edu/grafana/d/asjkldfbnvajdsfh213/leviathan-cluster?orgId=1 Grafana Monitor]

== Storage ==
Each server's "/home" directory located on the department's NAS. This means your files will be accessible on every server.

The NAS is a total of 100TB. This space is shared and monitored for fair usage. If you store large amounts of data, you may be asked if it is all necessary.
To see available space on the NAS, run the command below in a terminal on the server.<syntaxhighlight lang="bash">
df -h | grep "/home"
</syntaxhighlight>To see your total usage run the command below. ''It may take a few minutes to finish''<syntaxhighlight lang="bash">
du -sh ~
</syntaxhighlight>

== Software ==
The servers within Statistics & Data Science offer a variety of software. Sudo access, also referred to as the ability to install software, on department servers is not allowed. If a software package you would like to use is not listed below, please contact remarks@stat.cmu.edu.
{| class="wikitable"
|+
!
!Hydras
!Ghidorahs
!Leviathans
|-
|Python
|X
|X
|X
|-
|R
|X
|X
|X
|-
|Apptainer
|X
|X
|X
|-
|Quarto
|X
|
|X
|-
|Gurobi
|X
|
|X
|-
|Rstudio-Server
|X
|
|X
|-
|Matlab
|X
|
|X
|-
|Julia
|X
|
|X
|}

Department Servers

2026-06-25T12:12:57Z

Jacobg2: /* Software */

== Hydras ==
The Hydras are virtual based serves hosted in the Campus Cloud. They offer slightly higher stability and are better for smaller computing jobs.

Hardware Specifications

* Intel Xeon CPU
** 12 cores @2.6Ghz
* 64GB of RAM
* NAS used for /home storage
Servers = Hydra1-4

ssh yourusername@hydra1.stat.cmu.edu

[https://info.stat.cmu.edu/grafana/d/beuw1q6y3dgjkc/hydra-cluster?orgId=1 Grafana Monitoring]
== Ghidorahs ==
The Ghidorahs are physical servers located throughout Baker Hall. They are designed for lighter GPU jobs.

Hardware Specifications

* Core i9-13900KF CPU
** 8 performance cores @5.4Ghz
** 16 Efficiency cores @4.3Ghz
* 64GB of RAM
* RTX 4090 24GB of video memory
* NAS used for /home storage

[https://info.stat.cmu.edu/grafana/d/edkar1rgy3l6od/ghidorah-cluster?orgId=1 Grafana Monitoring]
== Leviathans ==
The Leviathans are physical servers located throughout Baker Hall and the Cyert Data Center. They are designed for heavy CPU and RAM jobs.

Hardware Specifications

* AMD Ryzen 9 9950x
** 16 cores @5.7Ghz
* 128GB of RAM (Leviathan1-6)
* 196GB of RAM (Leviathan7-11)
* 4TB of local NVME storage
** Mounted at /scratch
** Data stored in this location will only be accessible on the original server.
* NAS used for /home storage
Servers = Leviathan1-11

ssh yourusername@leviathan1.stat.cmu.edu

* You can use "levi" in place of "leviathan"
** ''ssh yourusername@levi1.stat.cmu.edu''

[https://info.stat.cmu.edu/grafana/d/asjkldfbnvajdsfh213/leviathan-cluster?orgId=1 Grafana Monitor]

== Storage ==
Each server's "/home" directory located on the department's NAS. This means your files will be accessible on every server.

The NAS is a total of 100TB. This space is shared and monitored for fair usage. If you store large amounts of data, you may be asked if it is all necessary.
To see available space on the NAS, run the command below in a terminal on the server.<syntaxhighlight lang="bash">
df -h | grep "/home"
</syntaxhighlight>To see your total usage run the command below. ''It may take a few minutes to finish''<syntaxhighlight lang="bash">
du -sh ~
</syntaxhighlight>

== Using the Servers ==

Accessing the servers requires using an SSH compatible terminal. Applications such as "Command Prompt" on Windows or "Terminal" on MacOS can achieve this.
To SSH to a server, open either app listed previously and enter the SSH command shown below. Be sure to use your StatDS computing account.<syntaxhighlight lang="bash">
ssh statdsusername@leviathan11.stat.cmu.edu
</syntaxhighlight>In the line above, replace "hydra1" with the server you want to connect to.
{| class="wikitable"
|+
Servers
!Hydras
!Ghidorahs
!Leviathans
|-
|hydra1
|ghidorah1
|leviathan1
|-
|hydra2
|ghidorah2
|leviathan2
|-
|hydra3
|ghidorah3
|leviathan3
|-
|hydra4
|
|leviathan4
|-
|
|
|leviathan5
|-
|
|
|leviathan6
|-
|
|
|leviathan7
|-
|
|
|leviathan8
|-
|
|
|leviathan9
|-
|
|
|leviathan10
|-
|
|
|leviathan11
|}

== Software ==
The servers within Statistics & Data Science offer a variety of software. Sudo access, also referred to as the ability to install software, on department servers is not allowed. If a software package you would like to use is not listed below, please contact remarks@stat.cmu.edu.
{| class="wikitable"
|+
!
!Hydras
!Ghidorahs
!Leviathans
|-
|Python
|X
|X
|X
|-
|R
|X
|X
|X
|-
|Apptainer
|X
|X
|X
|-
|Quarto
|X
|
|X
|-
|Gurobi
|X
|
|X
|-
|Rstudio-Server
|X
|
|X
|-
|Matlab
|X
|
|X
|-
|Julia
|X
|
|X
|}

Globus

2026-06-22T14:03:07Z

Jacobg2:

Do you have a need to share files located on the StatDS computing environment with external users?

If so, follow the steps below to configure and use Globus.

== Initial Set Up ==
This section will cover signing into Globus and connecting to the StatDS collection. Globus requires a separate account that is tied to your AndrewID.

1- Log in using the Google option with your AndrewID.

* <nowiki>https://app.globus.org/</nowiki>

2- Add the collection.

2a- Go to “File Manager” > “Collection” > then use the search bar and type "''Carnegie Mellon University Statistics & Data Science"''
[[File:Globus1.png|thumb|1023x1023px|alt=|none]]

2b- Select the collection, then click “Continue” to sign in.
[[File:Globus2.png|none|thumb|794x794px]]
2c- Once you are signed in, you will be taken to a page with all of your /home directory contents.

== Create a Shareable Link for External User ==
''The receiving user needs a Globus account''

1- Navigate to a directory you would like to share within the Statistics & Data Science Collection.

2- In the menu on the right of the page select “Share”.
[[File:Globus3.png|thumb|677x677px|alt=|none]]

3- In the new page select “Add Guest Collection”.
[[File:Globus4.png|thumb|680x680px|alt=|none]]

4- Verify the "Directory" you want to share is correct and provide a “Name” and “Description”.
[[File:Globus5.png|thumb|699x699px|alt=|none]]

5 - Once the Collection is created, go to “Collections” on the left menu bar and select the newly created Collection.
[[File:Globus6.png|thumb|1041x1041px|alt=|none]]

6- Once in the Collection, select “Permissions” at the top and then “Add Permissions - Share With”.
[[File:Globus7.png|thumb|1041x1041px|alt=|none]]

7 - On the new page, your path will default to “/”. This is the root path of the folder you chose to share. Enter the email of the user whom you want to share with.

''Change permissions as required.''
[[File:Globus8.png|thumb|777x777px|alt=|none]]

The external user will receive an email that contains a link to access the collection with their GlobusID. The user can ONLY access the folder you created a collection for.[[File:Globus9.png|thumb|788x788px|alt=|none]]

Once the user opens Globus and the collection, they can download/upload (depending on the permissions you granted) to the collection.

''Screenshot below is of external user’s Globus''
[[File:Globus10.png|thumb|849x849px|alt=|none]]

Chrome Remote Desktop

2026-06-03T14:09:09Z

Jacobg2:

Follow the instructions below to install and use Chrome Remote Desktop.

== Download Chrome Remote Desktop ==
1. Go to https://remotedesktop.google.com/support and log in with your Andrew ID. Click the blue download button within "Share this Screen" to install the application.
[[File:RDP1.png|border|thumb|761x761px|alt=|none]]

2. A new window will pop-up, click "Add to Chrome" and follow the remaining prompts to download and install the application.
[[File:RDP2.png|thumb|738x738px|alt=|none]]

== Using Chrome Remote Desktop ==
1. Go to https://remotedesktop.google.com/support and click "Generate Code" to initiate a remote session. If you get a warning about allowing Chrome RDP to access certain settings on your machine, go through each prompted setting and make sure it has the proper access.
[[File:RDP5.png|none|thumb|749x749px]]

2. Share the code with the other party so they can control your machine.
[[File:RDP6.png|none|thumb|759x759px]]

Chrome Remote Desktop

2026-06-03T14:05:45Z

Jacobg2:

== Download Chrome Remote Desktop ==
Follow the instructions below to install and use Chrome Remote Desktop.

1. Go to https://remotedesktop.google.com/support and log in with your Andrew ID. Click the blue download within "Share this Screen" to install the application.
[[File:RDP1.png|border|thumb|761x761px|alt=|none]]

2. A new window will pop-up, click "Add to Chrome" and select any additional options to allow the download. Pay attention to the on screen instructions to finish installing the application.
[[File:RDP2.png|thumb|738x738px|alt=|none]]

== Using Chrome Remote Desktop ==
1. Go to https://remotedesktop.google.com/support and click "Generate Code" to initiate a remote session. If you get a warning about allowing Chrome RDP to access certain settings on your machine, go through each prompted setting and make sure it has the proper access.
[[File:RDP5.png|none|thumb|749x749px]]

2. Share the code with the other party so they can control your machine.
[[File:RDP6.png|none|thumb|759x759px]]

File:RDP6.png

2026-06-03T14:05:24Z

Jacobg2:

Share code

Chrome Remote Desktop

2026-06-03T14:01:38Z

Jacobg2: /* Using Chrome Remote Desktop */

File:RDP5.png

2026-06-03T14:01:15Z

Jacobg2:

Generate code

Chrome Remote Desktop

2026-06-03T13:55:20Z

Jacobg2:

Chrome Remote Desktop

2026-06-03T13:54:59Z

Jacobg2: /* Using Chrome Remote Desktop */

Chrome Remote Desktop

2026-06-03T13:54:08Z

Jacobg2: /* Using Chrome Remote Desktop */

Chrome Remote Desktop

2026-06-03T13:52:11Z

Jacobg2:

Chrome Remote Desktop

2026-06-03T13:51:38Z

Jacobg2:

Chrome Remote Desktop

2026-06-03T13:50:42Z

Jacobg2:

== Download the Application ==
Follow the instructions below to install and use Chrome Remote Desktop.

1. Go to https://remotedesktop.google.com/support and log in with your Andrew ID. Click the blue download within "Share this Screen" to install the application.
[[File:RDP1.png|border|left|thumb|724x724px]]

2. A new window will pop-up, click "Add to Chrome" and select any additional options to allow the download. Pay attention to the on screen instructions to finish installing the application.
[[File:RDP2.png|left|thumb|738x738px]]

Chrome Remote Desktop

2026-06-03T13:50:30Z

Jacobg2:

Chrome Remote Desktop

2026-06-03T13:46:19Z

Jacobg2:

== Downloading the Application ==
Follow the instructions below to install and use Chrome Remote Desktop.

1. Go to https://remotedesktop.google.com/support and log in with your Andrew ID. Click the blue download within "Share this Screen" to install the application.
[[File:RDP1.png|border|left|thumb|724x724px]]

2. A new window will pop-up, click "Add to Chrome" and select any additional options to allow the download.
[[File:RDP2.png|border|left|thumb|772x772px]]

3. Once the download completes, click "Accept & Install" under Ready to Install.
[[File:RDP3.png|border|left|thumb|800x800px]]

File:RDP3.png

2026-06-03T13:44:17Z

Jacobg2:

Accept and install

File:RDP2.png

2026-06-03T13:42:10Z

Jacobg2:

Download the app 2

Chrome Remote Desktop

2026-06-03T13:38:56Z

Jacobg2: