Stat DS Computing Wiki - User contributions [en]

Wright GPU Cluster

2026-10-23T14:22:16Z

Areinhar: /* Connecting and setting up your environment */ warn about no backups

The department has access to a powerful GPU-based cluster named [[wikipedia:Sewall_Wright|Wright]] which is hosted here at CMU. This is a 4-node and growing cluster that is a shared resource for Statistics & Data Science, Philosophy, and other departments in Dietrich College. Each department has priority access to its own compute node, but will be able to share the resources from other departments when not in use.

== Access requests ==
This resource is free to use to users in the Departments of Statistics & Data Science, Philosophy, and CMIST. These departments set policies determining which of their users are eligible. Users outside these departments must be sponsored by someone in the eligible departments, as part of a project or collaboration.

To request access, complete the [https://docs.google.com/forms/d/e/1FAIpQLSftCsplE06whGocn62DhwC6NpDSgZ-DMmUacrQID_slIfg6sA/viewform access request form] and our team will be in touch to discuss your request and onboard you.

== Cluster setup ==
The cluster currently consists of 4 servers:
*The head node: You log in to this to write code and run jobs. It has 128 GB of RAM, 16 cores, and '''no GPU'''.
*The compute nodes: Jobs run on these. Each has 256 CPU cores, 8 NVIDIA L40 GPU cards with 48 GB of video RAM each, approximately 1 TB of RAM, and a total of approximately 3 TB of local data/scratch space.
*The file server: Holds 50TB of storage. The head node and compute nodes connect via fiberoptic networking. Your home directory is hosted on the file server, so the same files are accessible on the head node and all compute nodes.
**Statistics & Data Science accounts are limited to 5TB of space. More can be requested if required.
The cluster is currently partitioned. One compute node is reserved for each department involved.

The file server has quotas enabled so that each department gets 1/3rd of the available storage. If a department user tries to add files beyond the quota, the writes will be rejected, and nobody in that department will be able to add files until space is freed. To check the quotas and your own usage, [https://util.stat.cmu.edu/quotareport/ view the quota report].

== Connecting and setting up your environment ==

=== SSH ===

From your terminal, type <code>ssh <YourStatID>@wright.hss.cmu.edu</code> . You can also avoid remembering this by adding the following snippet to your local <code>~/.ssh/config</code> file:
<pre>Host wright
Hostname wright.hss.cmu.edu
User YourStatID
</pre>

You can then connect by simply typing <code>ssh wright</code>.

This connects you to the '''head node.''' Here you can upload your data files, set up your scripts, and test things.

'''Do not run resource-intensive tasks directly on the head node'''. It doesn't have any GPUs, and it has fewer resources than the compute nodes. You can do quick tests to make sure your scripts are set up correctly, but submit all other jobs through Slurm.

=== Uploading files ===
To upload scripts or data files, you can use <code>scp</code> from your local computer (Mac or Linux); on Windows, you can get the Tectia SSH graphical client [https://www.cmu.edu/computing/services/endpoint/software/access.html free through CMU].

You can also connect through the [https://code.visualstudio.com/docs/remote/ssh Remote SSH feature of VS Code] if you like. This is useful to

*navigate the file system from a proper GUI and 
*modify code/files/folder in place if you are not familiar with the command line
Files uploaded to your home directory will be accessible on the head node and all compute nodes, so they can be freely used in compute jobs.

=== Danger! Files are not backed up ===
Wright's storage cluster is '''not backed up.''' You may lose data if a disk fails. Therefore:

* Keep your code backed up in another location (such as by pushing it to GitHub regularly)
* Do not use Wright for long-term storage of important data. Download local copies, push it to HuggingFace, or sync it to your Google Drive
* Consider using an online backup tool like [https://www.borgbackup.org/ Borg] or [https://www.tarsnap.com/ Tarsnap] to regularly back up your home directory

=== Using Conda ===
The head node has Anaconda installed for managing Python packages and system libraries; see [[Python virtual environments]] for information on how to use it (but note it's '''already installed''', so you don't need to install it from scratch!).

However, the compute nodes '''do not''' have conda, so your slurm jobs can't use <code>conda activate</code> or other familiar commands. Instead, set up your environment on the head node, activate it, install the packages you need, and so on. Then run <code>which python</code> to get the path to the Python executable in your environment. For instance:<syntaxhighlight lang="bash">
$ which python
/home/export/areinhar/.conda/envs/llm/bin/python
</syntaxhighlight>This is the path you'll use in your scripts (see below).

=== Open OnDemand ===

Open OnDemand is a graphical interface where you can run jobs, interactive applications, and manage files. More information can be found [[Open OnDemand Wright|here]].

== Job scheduling and prioritization ==

The GPU cluster manages jobs via [[Slurm]]. Slurm is a job management system that deals with job prioritization and the allocation of resources. On high-performance computers (e.g., PSC and our GPU cluster) Slurm helps allocate resources so multiple jobs can be run quickly. Users run programs by submitting scripts to the Slurm job scheduler, and the scheduler runs them when resources are available.
=== Queue status ===

To check what jobs are running, use <code>squeue</code>. The following options list more helpful information than the default (though you need a wide terminal to read it all):<syntaxhighlight lang="bash">
$ squeue -O "JobID:7,UserName:10,Partition:12,TimeUsed:12,ReasonList,MinCPUS:9,MinMemory:12,tres-per-node,StartTime,EndTime"
JOBID USER PARTITION TIME NODELIST(REASON) MIN_CPUS MIN_MEMORY TRES_PER_NODE START_TIME END_TIME
36229 ebensen all 0:00 (DependencyNeverSati16 128G N/A N/A N/A
36237 ajshen all 12:35:57 n02 2 64G gres:gpu:1 2026-01-26T21:55:23 2026-01-28T21:55:23
36152 msezgin all 1-02:32:04 n03 1 64G gres:gpu:1 2026-01-26T07:59:16 2026-01-28T07:59:16
36132 tsukiant cmist_condo 1-11:00:02 n03 16 8G gres:gpu:1 2026-01-25T23:31:18 2026-01-27T23:31:18
36100 tsukiant cmist_condo 1-17:32:39 n03 16 8G gres:gpu:1 2026-01-25T16:58:41 2026-01-27T16:58:41
36090 tsukiant cmist_condo 1-18:22:51 n03 16 8G gres:gpu:1 2026-01-25T16:08:29 2026-01-27T16:08:29
36247 jwalches statds 42:38 n01 40 15G gres:gpu:3 2026-01-27T09:48:42 2026-01-29T09:48:42
36051 xiangchs statds_condo7:05:24 n01 64 128G gres:gpu:4 2026-01-27T03:25:56 2026-01-29T03:25:56
</syntaxhighlight>

Here we see there are two jobs running on node 1, one on node 2, four on node 3, and one job that is not running because it was submitted with requirements that will not be met. The listed end times are based on the time limit specified when the job was submitted, so the jobs may end sooner. If you submit a job that is waiting for resources, the reason will be listed as <code>(Resources)</code> or <code>(Priority)</code>, and after a few minutes Slurm will list a start time for your job based on when other jobs will finish and free up resources.

To check what resources are currently in use, use <code>sinfo</code>:<syntaxhighlight lang="bash">
$ sinfo -O "NodeList:10,CPUsState:16,Memory:9,FreeMem:10,Gres:20,GresUsed:30"
NODELIST CPUS(A/I/O/T) MEMORY FREE_MEM GRES GRES_USED
n01 64/192/0/256 1000000 221943 gpu:nvidia-l40:8 gpu:nvidia-l40:8(IDX:0-7)
n02 32/224/0/256 1000000 225575 gpu:nvidia-l40:8 gpu:nvidia-l40:0(IDX:N/A)
</syntaxhighlight>This shows that node 1 (Statistics & Data Science) has 64 CPUs allocated and 192 idle, out of 256 total. There is 1TB of RAM, of which 222 GB is free. There are 8 NVIDIA L40 GPus, all 8 of which are in use. Hence if we submit a job to the statds partition requesting any GPUs, it will have to wait until some become free. Meanwhile, node 2 (Philosophy) has no GPUs in use.

=== Partitions ===
The cluster is divided into three main partitions:

* <code>statds</code> (Statistics & Data Science)
* <code>phil</code> (Philosophy)
* <code>cmist</code> (CMIST)

Jobs are submitted to a partition. '''The main partitions will only accept jobs from users in the corresponding department''' -- jobs submitted to the wrong partition will be rejected. Partitions correspond to nodes, so this ensures each department's jobs use their own node.

If you want to use other nodes, you can submit jobs to the <code>all</code> partition. This requests any available node from any partition, so your job will run on whichever node has resources available. However, your job will have lower priority, as described below; if your all job runs on the philosophy node, a user who submits to <code>phil</code> can preempt your job, causing it to be killed to make room for them, then restarted after their job is done. See Job Etiquette below for when to use <code>all</code>.

If you want to use a specific node within <code>all</code> you can specify the node within your submit script as shown below.<syntaxhighlight lang="batch">
#SBATCH --nodelist=n03
</syntaxhighlight>

=== Job prioritization ===
Slurm runs jobs based on the resources they request and what is available. If resources aren't available for your job, slurm will put it in the queue and wait until other jobs finish. However, it's not simply a first-in first-out queue -- Slurm attempts to arrange the jobs so every job can run, even if that means running them out of order. Partitions have different quality-of-service (QOS) values that affect their prioritization, and Slurm also adjusts each user's priority based on how many jobs they submit, so that one user can't hog the cluster by submitting dozens of jobs; if one user has many jobs waiting, theirs will be reduced in priority so other users can run jobs as well.

Examples:

* Someone submits Job A that requests 6 GPUs. Later, you submit Job B that requests 4 GPUs on the same partition. The compute node only has 8 GPUs, so Job B will be queued until Job A completes.
* Someone submits Job A that requests 6 GPUs. Later, you submit Job B that requests 2 GPUs. Job B can run immediately.
* You submit a job that requires all 8 GPUs on a node, but 4 are currently in use. Your job is put in the queue and scheduled to run when Slurm knows the current jobs will finish (because of their maximum time limit). Meanwhile, someone submits a job using 2 GPUs and a short time limit. Slurm lets it run immediately, because it knows it will be done before your job is scheduled to run. Other jobs may be put off until after yours runs, so yours can get the 8 GPUs it needs.
* You submit 50 jobs that each require 1 GPU. The node has only 8 GPUs, but several are in use. Several of your jobs run, but the others are queued for later. The fair-sharing system assigns them a lower priority, because you have run so many jobs, and jobs submitted by other users will run first.
* You submit Job A to the <code>all</code> partition. Someone else submits Job B to <code>statds</code>. The <code>all</code> partition has lower QOS, so Job B will get to use the StatDS node first, and Job A will either wait or use a different node.
* You submit Job A to the <code>all</code> partition. It begins to run on the philosophy node. Someone else submits Job B to <code>phil</code>, and there are insufficient resources to run both at the same time. Job A is killed, Job B runs, and Job A is placed back into the queue to be run when resources are available.
You can view the currently running or queued jobs by running <code>squeue</code> on the head node, as described above. The <code>NODELIST(REASON)</code> column will indicate which jobs are queued by giving special codes:

* <code>(Resources)</code> means there are not enough resources (CPUs, memory, or GPUs) available for the job. It will run once the resources are free.
* <code>(Priority)</code> means resources are available, but Slurm wants to run a higher-priority job first. For example, maybe a large job has been waiting for resources for two days before you submit your job; Slurm may make your job wait so the large job can start first.

=== Job etiquette ===
The cluster is shared between users and departments. Therefore:

* Request the minimum resources required for your job, so other users can access other resources.
* For ordinary one-off jobs, use your department's partition so you get top priority for your node.
* If you need to submit a large number of jobs and aren't in a hurry, consider using the <code>all</code> partition. Using <code>all</code> lets you get all available resources, but at lower priority, so other users can run jobs as well and you don't hog the cluster.
* Watch your disk space usage. With large deep models it is easy to use many terabytes of storage, and each department has a quota. [https://util.stat.cmu.edu/quotareport/ You can check the quota and your own usage here.]

=== Job limits ===
There is no practical limit to the number of jobs you can have in the queue at a time (there is a maximum size limit for job arrays that can be checked via <code>scontrol show config | grep MaxArraySize</code>. However, in order to prevent resource monopolization, '''no user can have more than 6 jobs running at once on the <code>statds</code> partition.''' If a user submits more than 6 jobs to the <code>statds</code> partition, the excess jobs will sit in the queue until one or more of the first 6 are completed. '''This limit does not apply to the <code>all</code> partition''', so we encourage users with a large number of jobs to use the <code>all</code> partition instead of the <code>statds</code> partition.

== Setting up jobs ==

=== Submitting batch jobs ===
The best way to run jobs is by writing a shell script specifying the Slurm options. To launch a batch job from a script you will need two files:

*the actual script in your favorite programming language that does whatever computation you want
*a bash script that requests the resources you need from Slurm and gives the commands to launch your job.
A typical bash script might look like this:<pre>#!/bin/bash
# specify partition to run in (statds, phil, or cmist)
#SBATCH --partition statds
#SBATCH --ntasks 1
#SBATCH --job-name=nameofyourjob

# number of CPUs you need:
#SBATCH --cpus-per-task 1

# amount of memory per CPU
# (total allocated is CPUs * memory)
#SBATCH --mem-per-cpu 4G

# number of GPUs you need
#SBATCH --gres=gpu:1

# maximum time limit for job, HH:MM:SS (up to 48 hours max, 48:00:00)
#SBATCH --time=01:00:00

# email you when job is finished or errors out:
#SBATCH --mail-user=YourAndrewID@andrew.cmu.edu
#SBATCH --mail-type=END,FAIL

## Choose one:

# to run a Python script in a Conda environment called ENV
srun /home/export/YOURANDREWID/.conda/envs/ENV/bin/python YourScript.py

# to run an R script:
srun Rscript YourScript.R
</pre>Adjust the number of CPUs, number of GPUs, memory usage, and time limit based on the needs of your script. Requesting more than you need may result in your job having to wait for resources to free up; requesting a long time you don't need will block slurm from scheduling other jobs for when yours will be done. (And if you request more resources than exist on the compute nodes, your job will be queued forever while it waits for resources to appear.)

'''Note:''' Slurm batch jobs are limited to a maximum duration of 48 hours. Jobs running longer will be killed.

To run the job, SSH into the head node and navigate to the directory with your scripts. Then run:<syntaxhighlight lang="bash">
sbatch example.sh
</syntaxhighlight>where <code>example.sh</code> is the name of your bash script.

You will see a message like "Submitted batch job 1209" when your job is submitted. Any printed output from your script will automatically be redirected to a file based on the job number -- in this case, <code>slurm-1209.out</code>. You can use <code>tail -f slurm-1209.out</code> to view the output live and monitor your script's progress.

=== Selecting the right resources ===
In the Slurm script above, you can choose the number of tasks, the number of GPUs, the memory per CPU, the number of GPUs, and the time limit. Choose these values wisely:

* You generally want '''1 task''' (<code>--ntasks 1</code>). Asking for multiple tasks will cause srun to run multiple copies of your script simultaneously. If you need to run the same script with 10 different parameter values, this can be useful: request 10 tasks, and read the <code>SLURM_PROCID</code> environment variable in your script. It will be numbered 0-9 (for 10 tasks), so you can use this to set the parameter values for each parallel run.
* Set the number of CPUs based on your script. If your script is entirely single-threaded, ask for 1. If you use parallel processing (such as with multiprocessing in Python or the parallel package in R), use a multithreaded package, or spawn many parallel programs in your job, request more.
* Request the minimum amount of memory (RAM) needed for your script. This is per-CPU, so requesting a lot of CPUs and RAM can easily mean you request more RAM than exists on the node.
* Start with 1 GPU, adding more if your job requires more GPU memory (VRAM) than is on one GPU, or if you explicitly write your script to use more. If you do not use GPUs, request 0 so you aren't allocated a GPU that nobody else can use while your job runs.
* Set the time limit a little longer than you expect your script to run. Slurm uses the time limit when scheduling, so always choosing the maximum (48 hours) will backfire: Slurm will assume your job will take a long time and will run other things first. If you know your job can only take an hour, setting, say, a 01:30:00 time limit will let Slurm fit your job in before other long jobs in the queue. But of course if you choose too low and your job exceeds the limit, it will be killed, so leave yourself some extra time.

=== Interactive mode ===

You can also run interactive sessions that allow you to, say, run an R or Python console and type commands and immediately view the output. From the head node, run a command like:<syntaxhighlight lang="bash">
# remember to edit to correct partition:
srun --partition statds --ntasks 1 \
--cpus-per-task 16 --mem-per-cpu 4G \
--gres=gpu:1 --time=01:00:00 --pty \
bash
</syntaxhighlight>This runs a job that opens a bash shell on the compute node, and connects you directly to this job. You can now run bash commands, open Python, or run R scripts. Note your interactive session has a time limit and will be killed when it elapses, so set the time limit carefully and choose the resource amounts in the command according to your needs.

Interactive sessions are limited to '''two hours maximum'''. You cannot request a longer time limit. This is because interactive sessions are inefficient, with the cluster spending most of its time waiting for you to type something instead of running code. For large tasks, submit a batch job.

=== Interactive via VS Code ===
VS Code supports SSH and tunneling, which allows it to connect to a server and run commands. This allows you to use the Jupyter Notebook support built into VS Code to interact with the GPU cluster. As this uses an interactive session, it's subject to the same '''two-hour time limit'''. Hence notebooks should be used only for exploration and debugging, not large jobs.

To use this, we need to connect to the head node, run an interactive Slurm session, and then connect VS Code to that Slurm session so it can use it. This takes a few extra steps.

==== Setup ====
Before you use this for the first time, do the following steps:

*SSH into Wright and run the following to install and unpack the VSCode CLI: 
<pre>curl -Lk 'https://code.visualstudio.com/sha/download?build=stable&os=cli-alpine-x64' --output vscode_cli.tar.gz
tar -xf vscode_cli.tar.gz
</pre>

*Add the <code>code</code> executable to your PATH by adding the following to your <code>.bashrc</code> file: <code>export PATH="~:$PATH"</code><pre>echo 'export PATH="$PATH:/path/to/code"' >> ~/.bashrc
</pre>
*In VSCode, install the <code>Remote - Tunnels</code> extension, then use the VSCode SSH feature to connect to Wright and make sure to install all the local VS Code extensions on the remote machine as well (e.g., Python, Jupyter, ...)

==== Running a session ====
*In the Wright headnode, launch a Slurm interactive session with <code>srun</code>, following the instructions above.

*Inside the interactive session, run <code>code tunnel</code> and follow the instructions. The first time the system will ask you to name the tunnel and authorize the process via an authenticator, I suggest using GitHub since it usually works quite smoothly. Subsequent tunnels might still ask you to authorize their creation by typing an alphanumerical sequence into a link that appears in the terminal. Once you’ve done that, the tunnel is successfully created.
*From VSCode, select <code>Connect to Tunnel</code>, then choose the authentication method you previously used and you will see the tunnel appear in the dropdown. Select it and you will land directly into the interactive session we just created
*You will now be able to run code and, e.g., Jupyter Notebooks using the compute node allocated by the Slurm interactive session. VS Code will be disconnected when the Slurm session ends, such as when it reaches its time limit.
Note that if you need to use a conda environment, you'll have to configure VS Code to use the Python executable you found above, since conda is not installed on the compute nodes.

=== SSH to Compute Nodes ===
The ability to SSH directly to compute nodes is disabled. You can run the command below to start a bash terminal on a compute node when running a batch job. This allows for the same functionality as SSH.<syntaxhighlight lang="bash">
srun --jobid=yourjobidhere --pty /bin/bash
</syntaxhighlight>

== Monitoring cluster usage ==
The [https://util.stat.cmu.edu/grafana/d/Oxed_c6Wz/nvidia-wright-gpu-metrics NVIDIA Wright GPU Metrics Dashboard] shows graphs of GPU usage across the cluster. At the top left, you can select which node to graph, so viewing your department's node will let you see how GPUs are being used.

If you're developing a long-running job, we recommend checking this dashboard while your job is running. It is easy to accidentally write PyTorch or Tensorflow code that '''doesn't actually use the GPU''', so check the graphs to see if the expected number of GPUs are in used based on your job's requirements and the other jobs listed in <code>squeue</code>. If they are not, get a terminal on the compute node as described above and run <code>nvidia-smi</code> to get GPU usage information, including which scripts are accessing the GPU. If your script doesn't show up, you can debug it to figure out why no GPUs are being used.

== Using scratch storage ==
Each compute node (but not the head node) has several terabytes of scratch space. This is fast SSD storage that is directly attached to the node, rather than being accessed over the network (like all files in <code>/home</code> are). This is useful if your jobs need to write large temporary files: reading and writing the scratch space is faster than reading and writing files in <code>/home</code>.

Your home directory contains a symlink to the scratch space at <code>~/scratch</code>. To use it, create a directory inside named after your username:
mkdir ~/scratch/yourusername
Then you can read and write files in that directory.

'''Warnings:'''

* The scratch space is meant for '''temporary''' storage during a job. It may be cleared out during cluster maintenance or if the disk is full.
* The scratch drives are much smaller than the huge networked disk used for <code>/home</code>, so be careful what you store in it, and regularly delete files you don't need.
* Each node has separate scratch disks, so data you write on one node is '''not''' available from any other node, or from the head node.

The scratch space is thus not for final model outputs or results. It is ideal for things like saving checkpoints while your job is running, dumping data to debug a job, saving output from one script to be immediately used by another script, etc.

== Citing Wright ==
If you use Wright in your research, please make sure to cite it with the text provided below...<syntaxhighlight lang="text">
"This research was supported through the computational resources and staff contributions to the Wright HPC GPU cluster at Carnegie Mellon's Dietrich College of Humanities and Social Sciences provisioned and supported by the Department of Statistics & Data Science, the Department of Philosophy, The Carnegie Mellon Institute for Strategy and Technology, and the Dietrich College Dean's Office."
</syntaxhighlight>

Wright GPU Cluster

2026-08-21T14:24:45Z

Areinhar: fix category link

[[Category:Wright Cluster]]

The department has access to a powerful GPU-based cluster named [https://en.wikipedia.org/wiki/Sewall_Wright Wright] which is hosted here at CMU. This is a 4-node and growing cluster that is a shared resource for Statistics & Data Science, Philosophy, and other departments in Dietrich College. Each department has priority access to its own compute node, but will be able to share the resources from other departments when not in use.

== Access requests ==
This resource is free to use to users in the Departments of Statistics & Data Science, Philosophy, and CMIST. These departments set policies determining which of their users are eligible. Users outside these departments must be sponsored by someone in the eligible departments, as part of a project or collaboration.

To request access, complete the [https://docs.google.com/forms/d/e/1FAIpQLSftCsplE06whGocn62DhwC6NpDSgZ-DMmUacrQID_slIfg6sA/viewform access request form] and our team will be in touch to discuss your request and onboard you.

== Cluster setup ==
The cluster currently consists of 4 servers:
*The head node: You log in to this to write code and run jobs. It has 128 GB of RAM, 16 cores, and '''no GPU'''.
*The compute nodes: Jobs run on these. Each has 256 CPU cores, 8 NVIDIA L40 GPU cards with 48 GB of video RAM each, approximately 1 TB of RAM, and a total of approximately 3 TB of local data/scratch space.
*The file server: Holds 50TB of storage. The head node and compute nodes connect via fiberoptic networking. Your home directory is hosted on the file server, so the same files are accessible on the head node and all compute nodes.
The cluster is currently partitioned. One compute node is reserved for each department involved.

The file server has quotas enabled so that each department gets 1/3rd of the available storage. If a department user tries to add files beyond the quota, the writes will be rejected, and nobody in that department will be able to add files until space is freed. To check the quotas and your own usage, [https://util.stat.cmu.edu/quotareport/ view the quota report].

== Connecting and setting up your environment ==

=== SSH ===

From your terminal, type <code>ssh <YourStatID>@wright.hss.cmu.edu</code> . You can also avoid remembering this by adding the following snippet to your local <code>~/.ssh/config</code> file:
<pre>Host wright
Hostname wright.hss.cmu.edu
User YourStatID
</pre>

You can then connect by simply typing <code>ssh wright</code>.

This connects you to the '''head node.''' Here you can upload your data files, set up your scripts, and test things.

'''Do not run resource-intensive tasks directly on the head node'''. It doesn't have any GPUs, and it has fewer resources than the compute nodes. You can do quick tests to make sure your scripts are set up correctly, but submit all other jobs through Slurm.

=== Uploading files ===
To upload scripts or data files, you can use <code>scp</code> from your local computer (Mac or Linux); on Windows, you can get the Tectia SSH graphical client [https://www.cmu.edu/computing/services/endpoint/software/access.html free through CMU].

You can also connect through the [https://code.visualstudio.com/docs/remote/ssh Remote SSH feature of VS Code] if you like. This is useful to 

*navigate the file system from a proper GUI and 
*modify code/files/folder in place if you are not familiar with the command line
Files uploaded to your home directory will be accessible on the head node and all compute nodes, so they can be freely used in compute jobs.

=== Using Conda ===
The head node has Anaconda installed for managing Python packages and system libraries; see [[Python virtual environments]] for information on how to use it (but note it's '''already installed''', so you don't need to install it from scratch!).

However, the compute nodes '''do not''' have conda, so your slurm jobs can't use <code>conda activate</code> or other familiar commands. Instead, set up your environment on the head node, activate it, install the packages you need, and so on. Then run <code>which python</code> to get the path to the Python executable in your environment. For instance:<syntaxhighlight lang="bash">
$ which python
/home/export/areinhar/.conda/envs/llm/bin/python
</syntaxhighlight>This is the path you'll use in your scripts (see below).

=== Open OnDemand ===

Open OnDemand is a graphical interface where you can run jobs, interactive applications, and manage files. More information can be found [[Open OnDemand Wright|here]].

== Job scheduling and prioritization ==

The GPU cluster manages jobs via [[Slurm]]. Slurm is a job management system that deals with job prioritization and the allocation of resources. On high-performance computers (e.g., PSC and our GPU cluster) Slurm helps allocate resources so multiple jobs can be run quickly. Users run programs by submitting scripts to the Slurm job scheduler, and the scheduler runs them when resources are available.
=== Queue status ===

To check what jobs are running, use <code>squeue</code>. The following options list more helpful information than the default (though you need a wide terminal to read it all):<syntaxhighlight lang="bash">
$ squeue -O "JobID:7,UserName:10,Partition:12,TimeUsed:12,ReasonList,MinCPUS:9,MinMemory:12,tres-per-node,StartTime,EndTime"
JOBID USER PARTITION TIME NODELIST(REASON) MIN_CPUS MIN_MEMORY TRES_PER_NODE START_TIME END_TIME
36229 ebensen all 0:00 (DependencyNeverSati16 128G N/A N/A N/A
36237 ajshen all 12:35:57 n02 2 64G gres:gpu:1 2026-01-26T21:55:23 2026-01-28T21:55:23
36152 msezgin all 1-02:32:04 n03 1 64G gres:gpu:1 2026-01-26T07:59:16 2026-01-28T07:59:16
36132 tsukiant cmist_condo 1-11:00:02 n03 16 8G gres:gpu:1 2026-01-25T23:31:18 2026-01-27T23:31:18
36100 tsukiant cmist_condo 1-17:32:39 n03 16 8G gres:gpu:1 2026-01-25T16:58:41 2026-01-27T16:58:41
36090 tsukiant cmist_condo 1-18:22:51 n03 16 8G gres:gpu:1 2026-01-25T16:08:29 2026-01-27T16:08:29
36247 jwalches statds 42:38 n01 40 15G gres:gpu:3 2026-01-27T09:48:42 2026-01-29T09:48:42
36051 xiangchs statds_condo7:05:24 n01 64 128G gres:gpu:4 2026-01-27T03:25:56 2026-01-29T03:25:56
</syntaxhighlight>

Here we see there are two jobs running on node 1, one on node 2, four on node 3, and one job that is not running because it was submitted with requirements that will not be met. The listed end times are based on the time limit specified when the job was submitted, so the jobs may end sooner. If you submit a job that is waiting for resources, the reason will be listed as <code>(Resources)</code> or <code>(Priority)</code>, and after a few minutes Slurm will list a start time for your job based on when other jobs will finish and free up resources.

To check what resources are currently in use, use <code>sinfo</code>:<syntaxhighlight lang="bash">
$ sinfo -O "NodeList:10,CPUsState:16,Memory:9,FreeMem:10,Gres:20,GresUsed:30"
NODELIST CPUS(A/I/O/T) MEMORY FREE_MEM GRES GRES_USED
n01 64/192/0/256 1000000 221943 gpu:nvidia-l40:8 gpu:nvidia-l40:8(IDX:0-7)
n02 32/224/0/256 1000000 225575 gpu:nvidia-l40:8 gpu:nvidia-l40:0(IDX:N/A)
</syntaxhighlight>This shows that node 1 (Statistics & Data Science) has 64 CPUs allocated and 192 idle, out of 256 total. There is 1TB of RAM, of which 222 GB is free. There are 8 NVIDIA L40 GPus, all 8 of which are in use. Hence if we submit a job to the statds partition requesting any GPUs, it will have to wait until some become free. Meanwhile, node 2 (Philosophy) has no GPUs in use.

=== Partitions ===
The cluster is divided into three main partitions:

* <code>statds</code> (Statistics & Data Science)
* <code>phil</code> (Philosophy)
* <code>cmist</code> (CMIST)

Jobs are submitted to a partition. '''The main partitions will only accept jobs from users in the corresponding department''' -- jobs submitted to the wrong partition will be rejected. Partitions correspond to nodes, so this ensures each department's jobs use their own node.

If you want to use other nodes, you can submit jobs to the <code>all</code> partition. This requests any available node from any partition, so your job will run on whichever node has resources available. However, your job will have lower priority, as described below; if your all job runs on the philosophy node, a user who submits to <code>phil</code> can preempt your job, causing it to be killed to make room for them, then restarted after their job is done. See Job Etiquette below for when to use <code>all</code>.

If you want to use a specific node within <code>all</code> you can specify the node within your submit script as shown below.<syntaxhighlight lang="batch">
#SBATCH --nodelist=n03
</syntaxhighlight>

=== Job prioritization ===
Slurm runs jobs based on the resources they request and what is available. If resources aren't available for your job, slurm will put it in the queue and wait until other jobs finish. However, it's not simply a first-in first-out queue -- Slurm attempts to arrange the jobs so every job can run, even if that means running them out of order. Partitions have different quality-of-service (QOS) values that affect their prioritization, and Slurm also adjusts each user's priority based on how many jobs they submit, so that one user can't hog the cluster by submitting dozens of jobs; if one user has many jobs waiting, theirs will be reduced in priority so other users can run jobs as well.

Examples:

* Someone submits Job A that requests 6 GPUs. Later, you submit Job B that requests 4 GPUs on the same partition. The compute node only has 8 GPUs, so Job B will be queued until Job A completes.
* Someone submits Job A that requests 6 GPUs. Later, you submit Job B that requests 2 GPUs. Job B can run immediately.
* You submit a job that requires all 8 GPUs on a node, but 4 are currently in use. Your job is put in the queue and scheduled to run when Slurm knows the current jobs will finish (because of their maximum time limit). Meanwhile, someone submits a job using 2 GPUs and a short time limit. Slurm lets it run immediately, because it knows it will be done before your job is scheduled to run. Other jobs may be put off until after yours runs, so yours can get the 8 GPUs it needs.
* You submit 50 jobs that each require 1 GPU. The node has only 8 GPUs, but several are in use. Several of your jobs run, but the others are queued for later. The fair-sharing system assigns them a lower priority, because you have run so many jobs, and jobs submitted by other users will run first.
* You submit Job A to the <code>all</code> partition. Someone else submits Job B to <code>statds</code>. The <code>all</code> partition has lower QOS, so Job B will get to use the StatDS node first, and Job A will either wait or use a different node.
* You submit Job A to the <code>all</code> partition. It begins to run on the philosophy node. Someone else submits Job B to <code>phil</code>, and there are insufficient resources to run both at the same time. Job A is killed, Job B runs, and Job A is placed back into the queue to be run when resources are available.
You can view the currently running or queued jobs by running <code>squeue</code> on the head node, as described above. The <code>NODELIST(REASON)</code> column will indicate which jobs are queued by giving special codes:

* <code>(Resources)</code> means there are not enough resources (CPUs, memory, or GPUs) available for the job. It will run once the resources are free.
* <code>(Priority)</code> means resources are available, but Slurm wants to run a higher-priority job first. For example, maybe a large job has been waiting for resources for two days before you submit your job; Slurm may make your job wait so the large job can start first.

=== Job etiquette ===
The cluster is shared between users and departments. Therefore:

* Request the minimum resources required for your job, so other users can access other resources.
* For ordinary one-off jobs, use your department's partition so you get top priority for your node.
* If you need to submit a large number of jobs and aren't in a hurry, consider using the <code>all</code> partition. Using <code>all</code> lets you get all available resources, but at lower priority, so other users can run jobs as well and you don't hog the cluster.
* Watch your disk space usage. With large deep models it is easy to use many terabytes of storage, and each department has a quota. [https://util.stat.cmu.edu/quotareport/ You can check the quota and your own usage here.]

=== Job limits ===
There is no practical limit to the number of jobs you can have in the queue at a time (there is a maximum size limit for job arrays that can be checked via <code>scontrol show config | grep MaxArraySize</code>. However, in order to prevent resource monopolization, '''no user can have more than 6 jobs running at once on the <code>statds</code> partition.''' If a user submits more than 6 jobs to the <code>statds</code> partition, the excess jobs will sit in the queue until one or more of the first 6 are completed. '''This limit does not apply to the <code>all</code> partition''', so we encourage users with a large number of jobs to use the <code>all</code> partition instead of the <code>statds</code> partition.

== Setting up jobs ==

=== Submitting batch jobs ===
The best way to run jobs is by writing a shell script specifying the Slurm options. To launch a batch job from a script you will need two files:

*the actual script in your favorite programming language that does whatever computation you want
*a bash script that requests the resources you need from Slurm and gives the commands to launch your job.
A typical bash script might look like this:<pre>#!/bin/bash
# specify partition to run in (statds, phil, or cmist)
#SBATCH --partition statds
#SBATCH --ntasks 1
#SBATCH --job-name=nameofyourjob

# number of CPUs you need:
#SBATCH --cpus-per-task 1

# amount of memory per CPU
# (total allocated is CPUs * memory)
#SBATCH --mem-per-cpu 4G

# number of GPUs you need
#SBATCH --gres=gpu:1

# maximum time limit for job, HH:MM:SS (up to 48 hours max, 48:00:00)
#SBATCH --time=01:00:00

# email you when job is finished or errors out:
#SBATCH --mail-user=YourAndrewID@andrew.cmu.edu
#SBATCH --mail-type=END,FAIL

## Choose one:

# to run a Python script in a Conda environment called ENV
srun /home/export/YOURANDREWID/.conda/envs/ENV/bin/python YourScript.py

# to run an R script:
srun Rscript YourScript.R
</pre>Adjust the number of CPUs, number of GPUs, memory usage, and time limit based on the needs of your script. Requesting more than you need may result in your job having to wait for resources to free up; requesting a long time you don't need will block slurm from scheduling other jobs for when yours will be done. (And if you request more resources than exist on the compute nodes, your job will be queued forever while it waits for resources to appear.)

'''Note:''' Slurm batch jobs are limited to a maximum duration of 48 hours. Jobs running longer will be killed.

To run the job, SSH into the head node and navigate to the directory with your scripts. Then run:<syntaxhighlight lang="bash">
sbatch example.sh
</syntaxhighlight>where <code>example.sh</code> is the name of your bash script.

You will see a message like "Submitted batch job 1209" when your job is submitted. Any printed output from your script will automatically be redirected to a file based on the job number -- in this case, <code>slurm-1209.out</code>. You can use <code>tail -f slurm-1209.out</code> to view the output live and monitor your script's progress.

=== Selecting the right resources ===
In the Slurm script above, you can choose the number of tasks, the number of GPUs, the memory per CPU, the number of GPUs, and the time limit. Choose these values wisely:

* You generally want '''1 task''' (<code>--ntasks 1</code>). Asking for multiple tasks will cause srun to run multiple copies of your script simultaneously. If you need to run the same script with 10 different parameter values, this can be useful: request 10 tasks, and read the <code>SLURM_PROCID</code> environment variable in your script. It will be numbered 0-9 (for 10 tasks), so you can use this to set the parameter values for each parallel run.
* Set the number of CPUs based on your script. If your script is entirely single-threaded, ask for 1. If you use parallel processing (such as with multiprocessing in Python or the parallel package in R), use a multithreaded package, or spawn many parallel programs in your job, request more.
* Request the minimum amount of memory (RAM) needed for your script. This is per-CPU, so requesting a lot of CPUs and RAM can easily mean you request more RAM than exists on the node.
* Start with 1 GPU, adding more if your job requires more GPU memory (VRAM) than is on one GPU, or if you explicitly write your script to use more. If you do not use GPUs, request 0 so you aren't allocated a GPU that nobody else can use while your job runs.
* Set the time limit a little longer than you expect your script to run. Slurm uses the time limit when scheduling, so always choosing the maximum (48 hours) will backfire: Slurm will assume your job will take a long time and will run other things first. If you know your job can only take an hour, setting, say, a 01:30:00 time limit will let Slurm fit your job in before other long jobs in the queue. But of course if you choose too low and your job exceeds the limit, it will be killed, so leave yourself some extra time.

=== Interactive mode ===

You can also run interactive sessions that allow you to, say, run an R or Python console and type commands and immediately view the output. From the head node, run a command like:<syntaxhighlight lang="bash">
# remember to edit to correct partition:
srun --partition statds --ntasks 1 \
--cpus-per-task 16 --mem-per-cpu 4G \
--gres=gpu:1 --time=01:00:00 --pty \
bash
</syntaxhighlight>This runs a job that opens a bash shell on the compute node, and connects you directly to this job. You can now run bash commands, open Python, or run R scripts. Note your interactive session has a time limit and will be killed when it elapses, so set the time limit carefully and choose the resource amounts in the command according to your needs.

Interactive sessions are limited to '''two hours maximum'''. You cannot request a longer time limit. This is because interactive sessions are inefficient, with the cluster spending most of its time waiting for you to type something instead of running code. For large tasks, submit a batch job.

=== Interactive via VS Code ===
VS Code supports SSH and tunneling, which allows it to connect to a server and run commands. This allows you to use the Jupyter Notebook support built into VS Code to interact with the GPU cluster. As this uses an interactive session, it's subject to the same '''two-hour time limit'''. Hence notebooks should be used only for exploration and debugging, not large jobs.

To use this, we need to connect to the head node, run an interactive Slurm session, and then connect VS Code to that Slurm session so it can use it. This takes a few extra steps.

==== Setup ====
Before you use this for the first time, do the following steps:

*SSH into Wright and run the following to install and unpack the VSCode CLI: 
<pre>curl -Lk 'https://code.visualstudio.com/sha/download?build=stable&os=cli-alpine-x64' --output vscode_cli.tar.gz
tar -xf vscode_cli.tar.gz
</pre>

*Add the <code>code</code> executable to your PATH by adding the following to your <code>.bashrc</code> file: <code>export PATH="~:$PATH"</code><pre>echo 'export PATH="$PATH:/path/to/code"' >> ~/.bashrc
</pre>
*In VSCode, install the <code>Remote - Tunnels</code> extension, then use the VSCode SSH feature to connect to Wright and make sure to install all the local VS Code extensions on the remote machine as well (e.g., Python, Jupyter, ...)

==== Running a session ====
*In the Wright headnode, launch a Slurm interactive session with <code>srun</code>, following the instructions above.

*Inside the interactive session, run <code>code tunnel</code> and follow the instructions. The first time the system will ask you to name the tunnel and authorize the process via an authenticator, I suggest using GitHub since it usually works quite smoothly. Subsequent tunnels might still ask you to authorize their creation by typing an alphanumerical sequence into a link that appears in the terminal. Once you’ve done that, the tunnel is successfully created.
*From VSCode, select <code>Connect to Tunnel</code>, then choose the authentication method you previously used and you will see the tunnel appear in the dropdown. Select it and you will land directly into the interactive session we just created
*You will now be able to run code and, e.g., Jupyter Notebooks using the compute node allocated by the Slurm interactive session. VS Code will be disconnected when the Slurm session ends, such as when it reaches its time limit.
Note that if you need to use a conda environment, you'll have to configure VS Code to use the Python executable you found above, since conda is not installed on the compute nodes.

=== SSH to Compute Nodes ===
The ability to SSH directly to compute nodes is disabled. You can run the command below to start a bash terminal on a compute node when running a batch job. This allows for the same functionality as SSH.<syntaxhighlight lang="bash">
srun --jobid=yourjobidhere --pty /bin/bash
</syntaxhighlight>

== Monitoring cluster usage ==
The [https://util.stat.cmu.edu/grafana/d/Oxed_c6Wz/nvidia-wright-gpu-metrics NVIDIA Wright GPU Metrics Dashboard] shows graphs of GPU usage across the cluster. At the top left, you can select which node to graph, so viewing your department's node will let you see how GPUs are being used.

If you're developing a long-running job, we recommend checking this dashboard while your job is running. It is easy to accidentally write PyTorch or Tensorflow code that '''doesn't actually use the GPU''', so check the graphs to see if the expected number of GPUs are in used based on your job's requirements and the other jobs listed in <code>squeue</code>. If they are not, get a terminal on the compute node as described above and run <code>nvidia-smi</code> to get GPU usage information, including which scripts are accessing the GPU. If your script doesn't show up, you can debug it to figure out why no GPUs are being used.

== Using scratch storage ==
Each compute node (but not the head node) has several terabytes of scratch space. This is fast SSD storage that is directly attached to the node, rather than being accessed over the network (like all files in <code>/home</code> are). This is useful if your jobs need to write large temporary files: reading and writing the scratch space is faster than reading and writing files in <code>/home</code>.

Your home directory contains a symlink to the scratch space at <code>~/scratch</code>. To use it, create a directory inside named after your username:
mkdir ~/scratch/yourusername
Then you can read and write files in that directory.

'''Warnings:'''

* The scratch space is meant for '''temporary''' storage during a job. It may be cleared out during cluster maintenance or if the disk is full.
* The scratch drives are much smaller than the huge networked disk used for <code>/home</code>, so be careful what you store in it, and regularly delete files you don't need.
* Each node has separate scratch disks, so data you write on one node is '''not''' available from any other node, or from the head node.

The scratch space is thus not for final model outputs or results. It is ideal for things like saving checkpoints while your job is running, dumping data to debug a job, saving output from one script to be immediately used by another script, etc.

Wright GPU Cluster

2026-08-21T14:23:59Z

Areinhar: /* Job etiquette */ clarify prioritization

[[index.php?title=Category:Wright Cluster]]

The department has access to a powerful GPU-based cluster named [https://en.wikipedia.org/wiki/Sewall_Wright Wright] which is hosted here at CMU. This is a 4-node and growing cluster that is a shared resource for Statistics & Data Science, Philosophy, and other departments in Dietrich College. Each department has priority access to its own compute node, but will be able to share the resources from other departments when not in use.

== Access requests ==
This resource is free to use to users in the Departments of Statistics & Data Science, Philosophy, and CMIST. These departments set policies determining which of their users are eligible. Users outside these departments must be sponsored by someone in the eligible departments, as part of a project or collaboration.

To request access, complete the [https://docs.google.com/forms/d/e/1FAIpQLSftCsplE06whGocn62DhwC6NpDSgZ-DMmUacrQID_slIfg6sA/viewform access request form] and our team will be in touch to discuss your request and onboard you.

== Cluster setup ==
The cluster currently consists of 4 servers:
*The head node: You log in to this to write code and run jobs. It has 128 GB of RAM, 16 cores, and '''no GPU'''.
*The compute nodes: Jobs run on these. Each has 256 CPU cores, 8 NVIDIA L40 GPU cards with 48 GB of video RAM each, approximately 1 TB of RAM, and a total of approximately 3 TB of local data/scratch space.
*The file server: Holds 50TB of storage. The head node and compute nodes connect via fiberoptic networking. Your home directory is hosted on the file server, so the same files are accessible on the head node and all compute nodes.
The cluster is currently partitioned. One compute node is reserved for each department involved.

The file server has quotas enabled so that each department gets 1/3rd of the available storage. If a department user tries to add files beyond the quota, the writes will be rejected, and nobody in that department will be able to add files until space is freed. To check the quotas and your own usage, [https://util.stat.cmu.edu/quotareport/ view the quota report].

== Connecting and setting up your environment ==

=== SSH ===

From your terminal, type <code>ssh <YourStatID>@wright.hss.cmu.edu</code> . You can also avoid remembering this by adding the following snippet to your local <code>~/.ssh/config</code> file:
<pre>Host wright
Hostname wright.hss.cmu.edu
User YourStatID
</pre>

You can then connect by simply typing <code>ssh wright</code>.

This connects you to the '''head node.''' Here you can upload your data files, set up your scripts, and test things.

'''Do not run resource-intensive tasks directly on the head node'''. It doesn't have any GPUs, and it has fewer resources than the compute nodes. You can do quick tests to make sure your scripts are set up correctly, but submit all other jobs through Slurm.

=== Uploading files ===
To upload scripts or data files, you can use <code>scp</code> from your local computer (Mac or Linux); on Windows, you can get the Tectia SSH graphical client [https://www.cmu.edu/computing/services/endpoint/software/access.html free through CMU].

You can also connect through the [https://code.visualstudio.com/docs/remote/ssh Remote SSH feature of VS Code] if you like. This is useful to 

*navigate the file system from a proper GUI and 
*modify code/files/folder in place if you are not familiar with the command line
Files uploaded to your home directory will be accessible on the head node and all compute nodes, so they can be freely used in compute jobs.

=== Using Conda ===
The head node has Anaconda installed for managing Python packages and system libraries; see [[Python virtual environments]] for information on how to use it (but note it's '''already installed''', so you don't need to install it from scratch!).

However, the compute nodes '''do not''' have conda, so your slurm jobs can't use <code>conda activate</code> or other familiar commands. Instead, set up your environment on the head node, activate it, install the packages you need, and so on. Then run <code>which python</code> to get the path to the Python executable in your environment. For instance:<syntaxhighlight lang="bash">
$ which python
/home/export/areinhar/.conda/envs/llm/bin/python
</syntaxhighlight>This is the path you'll use in your scripts (see below).

=== Open OnDemand ===

Open OnDemand is a graphical interface where you can run jobs, interactive applications, and manage files. More information can be found [[Open OnDemand Wright|here]].

== Job scheduling and prioritization ==

The GPU cluster manages jobs via [[Slurm]]. Slurm is a job management system that deals with job prioritization and the allocation of resources. On high-performance computers (e.g., PSC and our GPU cluster) Slurm helps allocate resources so multiple jobs can be run quickly. Users run programs by submitting scripts to the Slurm job scheduler, and the scheduler runs them when resources are available.
=== Queue status ===

To check what jobs are running, use <code>squeue</code>. The following options list more helpful information than the default (though you need a wide terminal to read it all):<syntaxhighlight lang="bash">
$ squeue -O "JobID:7,UserName:10,Partition:12,TimeUsed:12,ReasonList,MinCPUS:9,MinMemory:12,tres-per-node,StartTime,EndTime"
JOBID USER PARTITION TIME NODELIST(REASON) MIN_CPUS MIN_MEMORY TRES_PER_NODE START_TIME END_TIME
36229 ebensen all 0:00 (DependencyNeverSati16 128G N/A N/A N/A
36237 ajshen all 12:35:57 n02 2 64G gres:gpu:1 2026-01-26T21:55:23 2026-01-28T21:55:23
36152 msezgin all 1-02:32:04 n03 1 64G gres:gpu:1 2026-01-26T07:59:16 2026-01-28T07:59:16
36132 tsukiant cmist_condo 1-11:00:02 n03 16 8G gres:gpu:1 2026-01-25T23:31:18 2026-01-27T23:31:18
36100 tsukiant cmist_condo 1-17:32:39 n03 16 8G gres:gpu:1 2026-01-25T16:58:41 2026-01-27T16:58:41
36090 tsukiant cmist_condo 1-18:22:51 n03 16 8G gres:gpu:1 2026-01-25T16:08:29 2026-01-27T16:08:29
36247 jwalches statds 42:38 n01 40 15G gres:gpu:3 2026-01-27T09:48:42 2026-01-29T09:48:42
36051 xiangchs statds_condo7:05:24 n01 64 128G gres:gpu:4 2026-01-27T03:25:56 2026-01-29T03:25:56
</syntaxhighlight>

Here we see there are two jobs running on node 1, one on node 2, four on node 3, and one job that is not running because it was submitted with requirements that will not be met. The listed end times are based on the time limit specified when the job was submitted, so the jobs may end sooner. If you submit a job that is waiting for resources, the reason will be listed as <code>(Resources)</code> or <code>(Priority)</code>, and after a few minutes Slurm will list a start time for your job based on when other jobs will finish and free up resources.

To check what resources are currently in use, use <code>sinfo</code>:<syntaxhighlight lang="bash">
$ sinfo -O "NodeList:10,CPUsState:16,Memory:9,FreeMem:10,Gres:20,GresUsed:30"
NODELIST CPUS(A/I/O/T) MEMORY FREE_MEM GRES GRES_USED
n01 64/192/0/256 1000000 221943 gpu:nvidia-l40:8 gpu:nvidia-l40:8(IDX:0-7)
n02 32/224/0/256 1000000 225575 gpu:nvidia-l40:8 gpu:nvidia-l40:0(IDX:N/A)
</syntaxhighlight>This shows that node 1 (Statistics & Data Science) has 64 CPUs allocated and 192 idle, out of 256 total. There is 1TB of RAM, of which 222 GB is free. There are 8 NVIDIA L40 GPus, all 8 of which are in use. Hence if we submit a job to the statds partition requesting any GPUs, it will have to wait until some become free. Meanwhile, node 2 (Philosophy) has no GPUs in use.

=== Partitions ===
The cluster is divided into three main partitions:

* <code>statds</code> (Statistics & Data Science)
* <code>phil</code> (Philosophy)
* <code>cmist</code> (CMIST)

Jobs are submitted to a partition. '''The main partitions will only accept jobs from users in the corresponding department''' -- jobs submitted to the wrong partition will be rejected. Partitions correspond to nodes, so this ensures each department's jobs use their own node.

If you want to use other nodes, you can submit jobs to the <code>all</code> partition. This requests any available node from any partition, so your job will run on whichever node has resources available. However, your job will have lower priority, as described below; if your all job runs on the philosophy node, a user who submits to <code>phil</code> can preempt your job, causing it to be killed to make room for them, then restarted after their job is done. See Job Etiquette below for when to use <code>all</code>.

If you want to use a specific node within <code>all</code> you can specify the node within your submit script as shown below.<syntaxhighlight lang="batch">
#SBATCH --nodelist=n03
</syntaxhighlight>

=== Job prioritization ===
Slurm runs jobs based on the resources they request and what is available. If resources aren't available for your job, slurm will put it in the queue and wait until other jobs finish. However, it's not simply a first-in first-out queue -- Slurm attempts to arrange the jobs so every job can run, even if that means running them out of order. Partitions have different quality-of-service (QOS) values that affect their prioritization, and Slurm also adjusts each user's priority based on how many jobs they submit, so that one user can't hog the cluster by submitting dozens of jobs; if one user has many jobs waiting, theirs will be reduced in priority so other users can run jobs as well.

Examples:

* Someone submits Job A that requests 6 GPUs. Later, you submit Job B that requests 4 GPUs on the same partition. The compute node only has 8 GPUs, so Job B will be queued until Job A completes.
* Someone submits Job A that requests 6 GPUs. Later, you submit Job B that requests 2 GPUs. Job B can run immediately.
* You submit a job that requires all 8 GPUs on a node, but 4 are currently in use. Your job is put in the queue and scheduled to run when Slurm knows the current jobs will finish (because of their maximum time limit). Meanwhile, someone submits a job using 2 GPUs and a short time limit. Slurm lets it run immediately, because it knows it will be done before your job is scheduled to run. Other jobs may be put off until after yours runs, so yours can get the 8 GPUs it needs.
* You submit 50 jobs that each require 1 GPU. The node has only 8 GPUs, but several are in use. Several of your jobs run, but the others are queued for later. The fair-sharing system assigns them a lower priority, because you have run so many jobs, and jobs submitted by other users will run first.
* You submit Job A to the <code>all</code> partition. Someone else submits Job B to <code>statds</code>. The <code>all</code> partition has lower QOS, so Job B will get to use the StatDS node first, and Job A will either wait or use a different node.
* You submit Job A to the <code>all</code> partition. It begins to run on the philosophy node. Someone else submits Job B to <code>phil</code>, and there are insufficient resources to run both at the same time. Job A is killed, Job B runs, and Job A is placed back into the queue to be run when resources are available.
You can view the currently running or queued jobs by running <code>squeue</code> on the head node, as described above. The <code>NODELIST(REASON)</code> column will indicate which jobs are queued by giving special codes:

* <code>(Resources)</code> means there are not enough resources (CPUs, memory, or GPUs) available for the job. It will run once the resources are free.
* <code>(Priority)</code> means resources are available, but Slurm wants to run a higher-priority job first. For example, maybe a large job has been waiting for resources for two days before you submit your job; Slurm may make your job wait so the large job can start first.

=== Job etiquette ===
The cluster is shared between users and departments. Therefore:

* Request the minimum resources required for your job, so other users can access other resources.
* For ordinary one-off jobs, use your department's partition so you get top priority for your node.
* If you need to submit a large number of jobs and aren't in a hurry, consider using the <code>all</code> partition. Using <code>all</code> lets you get all available resources, but at lower priority, so other users can run jobs as well and you don't hog the cluster.
* Watch your disk space usage. With large deep models it is easy to use many terabytes of storage, and each department has a quota. [https://util.stat.cmu.edu/quotareport/ You can check the quota and your own usage here.]

=== Job limits ===
There is no practical limit to the number of jobs you can have in the queue at a time (there is a maximum size limit for job arrays that can be checked via <code>scontrol show config | grep MaxArraySize</code>. However, in order to prevent resource monopolization, '''no user can have more than 6 jobs running at once on the <code>statds</code> partition.''' If a user submits more than 6 jobs to the <code>statds</code> partition, the excess jobs will sit in the queue until one or more of the first 6 are completed. '''This limit does not apply to the <code>all</code> partition''', so we encourage users with a large number of jobs to use the <code>all</code> partition instead of the <code>statds</code> partition.

== Setting up jobs ==

=== Submitting batch jobs ===
The best way to run jobs is by writing a shell script specifying the Slurm options. To launch a batch job from a script you will need two files:

*the actual script in your favorite programming language that does whatever computation you want
*a bash script that requests the resources you need from Slurm and gives the commands to launch your job.
A typical bash script might look like this:<pre>#!/bin/bash
# specify partition to run in (statds, phil, or cmist)
#SBATCH --partition statds
#SBATCH --ntasks 1
#SBATCH --job-name=nameofyourjob

# number of CPUs you need:
#SBATCH --cpus-per-task 1

# amount of memory per CPU
# (total allocated is CPUs * memory)
#SBATCH --mem-per-cpu 4G

# number of GPUs you need
#SBATCH --gres=gpu:1

# maximum time limit for job, HH:MM:SS (up to 48 hours max, 48:00:00)
#SBATCH --time=01:00:00

# email you when job is finished or errors out:
#SBATCH --mail-user=YourAndrewID@andrew.cmu.edu
#SBATCH --mail-type=END,FAIL

## Choose one:

# to run a Python script in a Conda environment called ENV
srun /home/export/YOURANDREWID/.conda/envs/ENV/bin/python YourScript.py

# to run an R script:
srun Rscript YourScript.R
</pre>Adjust the number of CPUs, number of GPUs, memory usage, and time limit based on the needs of your script. Requesting more than you need may result in your job having to wait for resources to free up; requesting a long time you don't need will block slurm from scheduling other jobs for when yours will be done. (And if you request more resources than exist on the compute nodes, your job will be queued forever while it waits for resources to appear.)

'''Note:''' Slurm batch jobs are limited to a maximum duration of 48 hours. Jobs running longer will be killed.

To run the job, SSH into the head node and navigate to the directory with your scripts. Then run:<syntaxhighlight lang="bash">
sbatch example.sh
</syntaxhighlight>where <code>example.sh</code> is the name of your bash script.

You will see a message like "Submitted batch job 1209" when your job is submitted. Any printed output from your script will automatically be redirected to a file based on the job number -- in this case, <code>slurm-1209.out</code>. You can use <code>tail -f slurm-1209.out</code> to view the output live and monitor your script's progress.

=== Selecting the right resources ===
In the Slurm script above, you can choose the number of tasks, the number of GPUs, the memory per CPU, the number of GPUs, and the time limit. Choose these values wisely:

* You generally want '''1 task''' (<code>--ntasks 1</code>). Asking for multiple tasks will cause srun to run multiple copies of your script simultaneously. If you need to run the same script with 10 different parameter values, this can be useful: request 10 tasks, and read the <code>SLURM_PROCID</code> environment variable in your script. It will be numbered 0-9 (for 10 tasks), so you can use this to set the parameter values for each parallel run.
* Set the number of CPUs based on your script. If your script is entirely single-threaded, ask for 1. If you use parallel processing (such as with multiprocessing in Python or the parallel package in R), use a multithreaded package, or spawn many parallel programs in your job, request more.
* Request the minimum amount of memory (RAM) needed for your script. This is per-CPU, so requesting a lot of CPUs and RAM can easily mean you request more RAM than exists on the node.
* Start with 1 GPU, adding more if your job requires more GPU memory (VRAM) than is on one GPU, or if you explicitly write your script to use more. If you do not use GPUs, request 0 so you aren't allocated a GPU that nobody else can use while your job runs.
* Set the time limit a little longer than you expect your script to run. Slurm uses the time limit when scheduling, so always choosing the maximum (48 hours) will backfire: Slurm will assume your job will take a long time and will run other things first. If you know your job can only take an hour, setting, say, a 01:30:00 time limit will let Slurm fit your job in before other long jobs in the queue. But of course if you choose too low and your job exceeds the limit, it will be killed, so leave yourself some extra time.

=== Interactive mode ===

You can also run interactive sessions that allow you to, say, run an R or Python console and type commands and immediately view the output. From the head node, run a command like:<syntaxhighlight lang="bash">
# remember to edit to correct partition:
srun --partition statds --ntasks 1 \
--cpus-per-task 16 --mem-per-cpu 4G \
--gres=gpu:1 --time=01:00:00 --pty \
bash
</syntaxhighlight>This runs a job that opens a bash shell on the compute node, and connects you directly to this job. You can now run bash commands, open Python, or run R scripts. Note your interactive session has a time limit and will be killed when it elapses, so set the time limit carefully and choose the resource amounts in the command according to your needs.

Interactive sessions are limited to '''two hours maximum'''. You cannot request a longer time limit. This is because interactive sessions are inefficient, with the cluster spending most of its time waiting for you to type something instead of running code. For large tasks, submit a batch job.

=== Interactive via VS Code ===
VS Code supports SSH and tunneling, which allows it to connect to a server and run commands. This allows you to use the Jupyter Notebook support built into VS Code to interact with the GPU cluster. As this uses an interactive session, it's subject to the same '''two-hour time limit'''. Hence notebooks should be used only for exploration and debugging, not large jobs.

To use this, we need to connect to the head node, run an interactive Slurm session, and then connect VS Code to that Slurm session so it can use it. This takes a few extra steps.

==== Setup ====
Before you use this for the first time, do the following steps:

*SSH into Wright and run the following to install and unpack the VSCode CLI: 
<pre>curl -Lk 'https://code.visualstudio.com/sha/download?build=stable&os=cli-alpine-x64' --output vscode_cli.tar.gz
tar -xf vscode_cli.tar.gz
</pre>

*Add the <code>code</code> executable to your PATH by adding the following to your <code>.bashrc</code> file: <code>export PATH="~:$PATH"</code><pre>echo 'export PATH="$PATH:/path/to/code"' >> ~/.bashrc
</pre>
*In VSCode, install the <code>Remote - Tunnels</code> extension, then use the VSCode SSH feature to connect to Wright and make sure to install all the local VS Code extensions on the remote machine as well (e.g., Python, Jupyter, ...)

==== Running a session ====
*In the Wright headnode, launch a Slurm interactive session with <code>srun</code>, following the instructions above.

*Inside the interactive session, run <code>code tunnel</code> and follow the instructions. The first time the system will ask you to name the tunnel and authorize the process via an authenticator, I suggest using GitHub since it usually works quite smoothly. Subsequent tunnels might still ask you to authorize their creation by typing an alphanumerical sequence into a link that appears in the terminal. Once you’ve done that, the tunnel is successfully created.
*From VSCode, select <code>Connect to Tunnel</code>, then choose the authentication method you previously used and you will see the tunnel appear in the dropdown. Select it and you will land directly into the interactive session we just created
*You will now be able to run code and, e.g., Jupyter Notebooks using the compute node allocated by the Slurm interactive session. VS Code will be disconnected when the Slurm session ends, such as when it reaches its time limit.
Note that if you need to use a conda environment, you'll have to configure VS Code to use the Python executable you found above, since conda is not installed on the compute nodes.

=== SSH to Compute Nodes ===
The ability to SSH directly to compute nodes is disabled. You can run the command below to start a bash terminal on a compute node when running a batch job. This allows for the same functionality as SSH.<syntaxhighlight lang="bash">
srun --jobid=yourjobidhere --pty /bin/bash
</syntaxhighlight>

== Monitoring cluster usage ==
The [https://util.stat.cmu.edu/grafana/d/Oxed_c6Wz/nvidia-wright-gpu-metrics NVIDIA Wright GPU Metrics Dashboard] shows graphs of GPU usage across the cluster. At the top left, you can select which node to graph, so viewing your department's node will let you see how GPUs are being used.

If you're developing a long-running job, we recommend checking this dashboard while your job is running. It is easy to accidentally write PyTorch or Tensorflow code that '''doesn't actually use the GPU''', so check the graphs to see if the expected number of GPUs are in used based on your job's requirements and the other jobs listed in <code>squeue</code>. If they are not, get a terminal on the compute node as described above and run <code>nvidia-smi</code> to get GPU usage information, including which scripts are accessing the GPU. If your script doesn't show up, you can debug it to figure out why no GPUs are being used.

== Using scratch storage ==
Each compute node (but not the head node) has several terabytes of scratch space. This is fast SSD storage that is directly attached to the node, rather than being accessed over the network (like all files in <code>/home</code> are). This is useful if your jobs need to write large temporary files: reading and writing the scratch space is faster than reading and writing files in <code>/home</code>.

Your home directory contains a symlink to the scratch space at <code>~/scratch</code>. To use it, create a directory inside named after your username:
mkdir ~/scratch/yourusername
Then you can read and write files in that directory.

'''Warnings:'''

* The scratch space is meant for '''temporary''' storage during a job. It may be cleared out during cluster maintenance or if the disk is full.
* The scratch drives are much smaller than the huge networked disk used for <code>/home</code>, so be careful what you store in it, and regularly delete files you don't need.
* Each node has separate scratch disks, so data you write on one node is '''not''' available from any other node, or from the head node.

The scratch space is thus not for final model outputs or results. It is ideal for things like saving checkpoints while your job is running, dumping data to debug a job, saving output from one script to be immediately used by another script, etc.

What resources when

2026-07-07T15:03:39Z

Areinhar:

[[Category:Department Servers]]
[[Category:Wright Cluster]]
[[Category:PSC]]
[[Category:Remote Server Access]]
To try to answer the common question “what resources do I have access to and when should I use them”, below we present to approaches to answering the question. The first of which focuses on what resources exist and what they should be used for and the second tries to answer resource needs / current computing problems with resources to fit those needs/ solve those problems.

After you’re found the resource you think you’d like to use (or even when you’re still deciding), we strongly encourage you to attend an computing office hour to help discuss all the things you should be thinking about and get you on the right track quickly.

== What computing resources do you have access to ==

=== Your personal computer ===

Your personal computer is great for small jobs that take less than 30 minutes, uses few cores and not too much RAM.

'''Pro:'''

#Naturally the easiest tool to use since it’s yours.
#Install whatever you want.
#Nobody else is hogging it.

'''Cons:'''

#

* capacity:
**few cores: 4-8 cores common on laptops (hyperthreading means 8-16 “cores”).
**low RAM: 4-32 GB of common on laptops, used across all cores
**“no” GPUs: you don’t have GPUs that are optimized for Deep Learning
*You're using your laptop for other things, and running a big job will slow everything down when you're trying to do your homework
*If your job lasts many hours, you may have to leave your laptop at home or leave it running overnight; or you may have problems when you close your laptop to take the bus.

[[File:Compiling.png|thumb|This does not have to be you (though it's good to take breaks) https://xkcd.com/303/|alt=]]

'''Conclusion:''' Your personal computer isn’t the best for when doing a lot of computing and its easy to put jobs elsewhere!

'''Common use-cases'''

#Most class projects
#Non-data problems
#Non-complex structure problems

'''Associated Articles''':

*[[Parallel_computing|parallelization]]

=== The department servers ===

The departmental servers should be seen as a “sandbox”. You can experiment with lots of things on the servers and it doesn’t cost you much to do. The Leviathans and Hydras have more RAM than a typical laptop. The Leviathans also have top-of-the-line CPUs that should be faster than most laptops, though the Hydras have older CPUs that may not be faster than running code on your own laptop.

'''Pros:'''

#Best for long-running jobs
#Plenty of RAM per machine (128 GB on Leviathans + hydra7 and 8, 64 GB on other hydras)
#Very low barrier to entry (aka easy to get started)
#File management system shares across all servers

'''Cons:'''

#usage: The department server is a bit of a “wild-west” in terms of usage (it varies how many people use it at a given time)
#cores per server: the number of cores per server is only 12 (Hydras) or 16 (Leviathans)

'''Conclusion:''' Great place to start for more intensive computing.

'''Associated Articles''':

*[[Linux_and_department_servers|Linux and department servers]]
'''Server Monitoring'''

* [https://info.stat.cmu.edu/grafana/d/beuw1q6y3dgjkc/hydra-cluster?orgId=1 Hydra Cluster Usage Monitor]
* [https://info.stat.cmu.edu/grafana/d/asjkldfbnvajdsfh213/leviathan-cluster?orgId=1 Leviathan Cluster Usage Monitor]

== PSC (Pittsburgh supercomputer center) ==

The Pittsburgh supercomputer center provides supercomputers than have massive amounts of cores (but also can find some with large RAMs and GPUs). Generally speaking, the supercomputing resources are often used when you have extremely parallelizable code that would take a very long time if you ran it on just a few cores (e.g. your computer or a few hydras).

'''Pros:'''

#A massive amount of cores and RAM that can be leveraged for extremely parallelizable computing
#Easier to access in the statistics department

'''Cons:'''

#Requires learning of new job management system (though quick to learn - see [Slurm])
#Some options to access the resources can take time or require going through your advisor
#''currently (2021)'' has been having problems with servers crashing

'''Associated Articles''':

*[[PSC|PSC]]
*[[Slurm|Slurm]]

== GPU Computing ==
The Statistics & Data Science Department offers two main GPU computing environments.

'''Ghidorahs'''

* 3 physical servers
* RTX 4090 GPUs
* Connected to the Stats network share
* Your statistics directory account will grant you access

'''Wright'''

* For a detail overview, please check [[GPU Cluster|here]]
* You must request access
* The Statistics network share is not accessible

== Computing resource flowchart ==

These flow charts try to help capture what’s available and when you might use them.

Visit the links below to view current computing resource usage. Select a server with the least amount of users and lowest resource usage.

* [https://info.stat.cmu.edu/grafana/d/fdkb6l4klj400d/hydra-cluster?orgId=1 Hydra Cluster Usage Monitor]

* [https://info.stat.cmu.edu/grafana/d/edkar1rgy3l6od/ghidorah-cluster?orgId=1 Ghidorah Cluster Usage Monitor]
[[File:ProblemsOnYourPersonalComputerUpdated.png|center|thumb|1000x1000px]]

From [http://laptopmag.com/articles/Cpu-comparison laptopmag]
<br/>      2. <font color="blue">↑</font>
From [http://digitaltrends.com/computing/how-much-ram digitaltrends]

What resources when

2026-07-07T15:03:28Z

Areinhar:

[[Category:Department Servers]]
[[Category:Wright Cluster]]
[[Category:PSC]]
[[Category:Remote Server Access]]

To try to answer the common question “what resources do I have access to and when should I use them”, below we present to approaches to answering the question. The first of which focuses on what resources exist and what they should be used for and the second tries to answer resource needs / current computing problems with resources to fit those needs/ solve those problems.

After you’re found the resource you think you’d like to use (or even when you’re still deciding), we strongly encourage you to attend an computing office hour to help discuss all the things you should be thinking about and get you on the right track quickly.

== What computing resources do you have access to ==

=== Your personal computer ===

Your personal computer is great for small jobs that take less than 30 minutes, uses few cores and not too much RAM.

'''Pro:'''

#Naturally the easiest tool to use since it’s yours.
#Install whatever you want.
#Nobody else is hogging it.

'''Cons:'''

#

* capacity:
**few cores: 4-8 cores common on laptops (hyperthreading means 8-16 “cores”).
**low RAM: 4-32 GB of common on laptops, used across all cores
**“no” GPUs: you don’t have GPUs that are optimized for Deep Learning
*You're using your laptop for other things, and running a big job will slow everything down when you're trying to do your homework
*If your job lasts many hours, you may have to leave your laptop at home or leave it running overnight; or you may have problems when you close your laptop to take the bus.

[[File:Compiling.png|thumb|This does not have to be you (though it's good to take breaks) https://xkcd.com/303/|alt=]]

'''Conclusion:''' Your personal computer isn’t the best for when doing a lot of computing and its easy to put jobs elsewhere!

'''Common use-cases'''

#Most class projects
#Non-data problems
#Non-complex structure problems

'''Associated Articles''':

*[[Parallel_computing|parallelization]]

=== The department servers ===

The departmental servers should be seen as a “sandbox”. You can experiment with lots of things on the servers and it doesn’t cost you much to do. The Leviathans and Hydras have more RAM than a typical laptop. The Leviathans also have top-of-the-line CPUs that should be faster than most laptops, though the Hydras have older CPUs that may not be faster than running code on your own laptop.

'''Pros:'''

#Best for long-running jobs
#Plenty of RAM per machine (128 GB on Leviathans + hydra7 and 8, 64 GB on other hydras)
#Very low barrier to entry (aka easy to get started)
#File management system shares across all servers

'''Cons:'''

#usage: The department server is a bit of a “wild-west” in terms of usage (it varies how many people use it at a given time)
#cores per server: the number of cores per server is only 12 (Hydras) or 16 (Leviathans)

'''Conclusion:''' Great place to start for more intensive computing.

'''Associated Articles''':

*[[Linux_and_department_servers|Linux and department servers]]
'''Server Monitoring'''

* [https://info.stat.cmu.edu/grafana/d/beuw1q6y3dgjkc/hydra-cluster?orgId=1 Hydra Cluster Usage Monitor]
* [https://info.stat.cmu.edu/grafana/d/asjkldfbnvajdsfh213/leviathan-cluster?orgId=1 Leviathan Cluster Usage Monitor]

== PSC (Pittsburgh supercomputer center) ==

The Pittsburgh supercomputer center provides supercomputers than have massive amounts of cores (but also can find some with large RAMs and GPUs). Generally speaking, the supercomputing resources are often used when you have extremely parallelizable code that would take a very long time if you ran it on just a few cores (e.g. your computer or a few hydras).

'''Pros:'''

#A massive amount of cores and RAM that can be leveraged for extremely parallelizable computing
#Easier to access in the statistics department

'''Cons:'''

#Requires learning of new job management system (though quick to learn - see [Slurm])
#Some options to access the resources can take time or require going through your advisor
#''currently (2021)'' has been having problems with servers crashing

'''Associated Articles''':

*[[PSC|PSC]]
*[[Slurm|Slurm]]

== GPU Computing ==
The Statistics & Data Science Department offers two main GPU computing environments.

'''Ghidorahs'''

* 3 physical servers
* RTX 4090 GPUs
* Connected to the Stats network share
* Your statistics directory account will grant you access

'''Wright'''

* For a detail overview, please check [[GPU Cluster|here]]
* You must request access
* The Statistics network share is not accessible

== Computing resource flowchart ==

These flow charts try to help capture what’s available and when you might use them.

Visit the links below to view current computing resource usage. Select a server with the least amount of users and lowest resource usage.

* [https://info.stat.cmu.edu/grafana/d/fdkb6l4klj400d/hydra-cluster?orgId=1 Hydra Cluster Usage Monitor]

* [https://info.stat.cmu.edu/grafana/d/edkar1rgy3l6od/ghidorah-cluster?orgId=1 Ghidorah Cluster Usage Monitor]
[[File:ProblemsOnYourPersonalComputerUpdated.png|center|thumb|1000x1000px]]

From [http://laptopmag.com/articles/Cpu-comparison laptopmag]
<br/>      2. <font color="blue">↑</font>
From [http://digitaltrends.com/computing/how-much-ram digitaltrends]

What resources when

2026-07-07T15:02:47Z

Areinhar: formatting

[[Category:Department Servers]]
[[Category:Wright Cluster]]
[[Category:PSC]]
[[Category:Remote Server Access]]

To try to answer the common question “what resources do I have access to and when should I use them”, below we present to approaches to answering the question. The first of which focuses on what resources exist and what they should be used for and the second tries to answer resource needs / current computing problems with resources to fit those needs/ solve those problems.

After you’re found the resource you think you’d like to use (or even when you’re still deciding), we strongly encourage you to attend an computing office hour to help discuss all the things you should be thinking about and get you on the right track quickly.

== What computing resources do you have access to ==

=== Your personal computer ===

Your personal computer is great for small jobs that take less than 30 minutes, uses few cores and not too much RAM.

'''Pro:'''

#Naturally the easiest tool to use since it’s yours.
#Install whatever you want.
#Nobody else is hogging it.

'''Cons:'''

#

* capacity:
**few cores: 4-8 cores common on laptops<ref><p>From [http://laptopmag.com/articles/Cpu-comparison laptopmag]</p></ref> (hyperthreading means 8-16 “cores”).
**low RAM: 4-32 GB of common on laptops<ref><p>From [http://digitaltrends.com/computing/how-much-ram digitaltrends]</p></ref>, used across all cores
**“no” GPUs: you don’t have GPUs that are optimized for Deep Learning
*You're using your laptop for other things, and running a big job will slow everything down when you're trying to do your homework
*If your job lasts many hours, you may have to leave your laptop at home or leave it running overnight; or you may have problems when you close your laptop to take the bus.

[[File:Compiling.png|thumb|This does not have to be you (though it's good to take breaks) https://xkcd.com/303/|alt=]]

'''Conclusion:''' Your personal computer isn’t the best for when doing a lot of computing and its easy to put jobs elsewhere!

'''Common use-cases'''

#Most class projects
#Non-data problems
#Non-complex structure problems

'''Associated Articles''':

*[[Parallel_computing|parallelization]]

=== The department servers ===

The departmental servers should be seen as a “sandbox”. You can experiment with lots of things on the servers and it doesn’t cost you much to do. The Leviathans and Hydras have more RAM than a typical laptop. The Leviathans also have top-of-the-line CPUs that should be faster than most laptops, though the Hydras have older CPUs that may not be faster than running code on your own laptop.

'''Pros:'''

#Best for long-running jobs
#Plenty of RAM per machine (128 GB on Leviathans + hydra7 and 8, 64 GB on other hydras)
#Very low barrier to entry (aka easy to get started)
#File management system shares across all servers

'''Cons:'''

#usage: The department server is a bit of a “wild-west” in terms of usage (it varies how many people use it at a given time)
#cores per server: the number of cores per server is only 12 (Hydras) or 16 (Leviathans)

'''Conclusion:''' Great place to start for more intensive computing.

'''Associated Articles''':

*[[Linux_and_department_servers|Linux and department servers]]
'''Server Monitoring'''

* [https://info.stat.cmu.edu/grafana/d/beuw1q6y3dgjkc/hydra-cluster?orgId=1 Hydra Cluster Usage Monitor]
* [https://info.stat.cmu.edu/grafana/d/asjkldfbnvajdsfh213/leviathan-cluster?orgId=1 Leviathan Cluster Usage Monitor]

== PSC (Pittsburgh supercomputer center) ==

The Pittsburgh supercomputer center provides supercomputers than have massive amounts of cores (but also can find some with large RAMs and GPUs). Generally speaking, the supercomputing resources are often used when you have extremely parallelizable code that would take a very long time if you ran it on just a few cores (e.g. your computer or a few hydras).

'''Pros:'''

#A massive amount of cores and RAM that can be leveraged for extremely parallelizable computing
#Easier to access in the statistics department

'''Cons:'''

#Requires learning of new job management system (though quick to learn - see [Slurm])
#Some options to access the resources can take time or require going through your advisor
#''currently (2021)'' has been having problems with servers crashing

'''Associated Articles''':

*[[PSC|PSC]]
*[[Slurm|Slurm]]

== GPU Computing ==
The Statistics & Data Science Department offers two main GPU computing environments.

'''Ghidorahs'''

* 3 physical servers
* RTX 4090 GPUs
* Connected to the Stats network share
* Your statistics directory account will grant you access

'''Wright'''

* For a detail overview, please check [[GPU Cluster|here]]
* You must request access
* The Statistics network share is not accessible

== Computing resource flowchart ==

These flow charts try to help capture what’s available and when you might use them.

Visit the links below to view current computing resource usage. Select a server with the least amount of users and lowest resource usage.

* [https://info.stat.cmu.edu/grafana/d/fdkb6l4klj400d/hydra-cluster?orgId=1 Hydra Cluster Usage Monitor]

* [https://info.stat.cmu.edu/grafana/d/edkar1rgy3l6od/ghidorah-cluster?orgId=1 Ghidorah Cluster Usage Monitor]
[[File:ProblemsOnYourPersonalComputerUpdated.png|center|thumb|1000x1000px]]

From [http://laptopmag.com/articles/Cpu-comparison laptopmag]
<br/>      2. <font color="blue">↑</font>
From [http://digitaltrends.com/computing/how-much-ram digitaltrends]

Building Workflows on Remote Servers

2026-07-07T14:25:41Z

Areinhar:

[[Category:Beginner's_Guide_Scaling|01]]
{{ScalingSidebar}}

Even with perfect parallelization, research experiments can quickly scale beyond the capabilities of any personal computing device. Thus, most workflows with high computing requirements are often performed on High-Performance Computing (HPC) clusters, which are dedicated servers for large computational workloads. These servers often have hardware orders of magnitude faster than personal devices.

Students in the CMU Department of Statistics and Data Science have access to several HPC clusters. Each cluster is configured for various types and sizes of workflows.

Back to [[Beginner's Guide to Computation in Statistical Research]]

== 1. Department Servers ==
The department wholly owns and operates a small HPC cluster, composed of several CPU-only nodes known as the Leviathans, and a few GPU nodes known as the Ghidorahs. The department cluster is suitable for modest sized workflows that would be too slow to run on a personal device.

<div style="display: grid; grid-template-columns: repeat(auto-fit, minmax(500px, 1fr))">
<div style="background-color: #fff3cd; border-left: 5px solid #ffc107; padding: 15px; margin-bottom: 20px; margin-top: 15px; overflow: hidden;">
'''Important: Resource Limits'''<br>
If your workflow requires continual, simultaneous use of multiple Leviathans or Ghidorahs, we strongly recommend migrating to the external shared servers in the next section to ensure fair use for all students.
</div>

<div style="background-color: #f8f9fa; border-left: 4px solid #6c757d; padding: 15px; margin-bottom: 10px;">
'''[[Linux and department servers]]'''<br>
Introduces basic Linux commands and instructions for connecting to the department servers. Start here for all server-based workflows.
</div>

<div style="background-color: #f8f9fa; border-left: 4px solid #75AADB; padding: 15px; margin-bottom: 10px;">
'''[[R on the server]]'''<br>
Demonstrates how to run R scripts and RStudio on the department servers.
</div>

<div style="background-color: #f8f9fa; border-left: 4px solid #005A9C; padding: 15px; margin-bottom: 10px;">
'''[[Python on the server]]'''<br>
Demonstrates how to run Python scripts and Jupyter notebooks on the department servers.
</div>
</div>

== 2. External Shared Servers ==
Students in the department also have access to other HPC resources that are shared between departments and institutions. These clusters are built for massive parallelization and GPU-intensive workloads.

<div style="display: grid; grid-template-columns: repeat(auto-fit, minmax(500px, 1fr))">
<div style="background-color: #f8f9fa; border-left: 4px solid #dc3545; padding: 15px; margin-bottom: 10px;">
'''[[Slurm]]'''<br>
Introduces the job scheduling system used on larger HPC clusters (like Wright and PSC). Essential reading for medium to large-scale server workflows.
</div>

<div style="background-color: #f8f9fa; border-left: 4px solid #28a745; padding: 15px; margin-bottom: 10px;">
'''[[GPU Cluster | Wright GPU Cluster]]'''<br>
The manual for accessing and using the Wright cluster. Ideal for large-scale, GPU-heavy server workflows.
</div>

<div style="background-color: #f8f9fa; border-left: 4px solid #6f42c1; padding: 15px; margin-bottom: 10px;">
'''[[PSC]] (Incomplete)'''<br>
Instructions for accessing and using the Pittsburgh Supercomputing Center (PSC) servers for massive-scale computing.
</div>
</div>

== 3. Other Useful Tools ==
Below are some useful tools and tricks for streamlining your work on any remote server.

<div style="background-color: #f8f9fa; border-left: 4px solid #17a2b8; padding: 15px; margin-bottom: 10px;">
'''[[VS Code Remote]]'''<br>
Introduces the VS Code Remote plugin, which allows you to edit files and run terminals using the full VS Code IDE experience directly on a remote server.
</div>

Python on the server

2026-07-07T14:25:17Z

Areinhar:

[[Category:IDEs]]
[[Category:Department Servers]]
[[Category:Wright Cluster]]
[[Category:PSC]]

There are multiple ways to run python on the server. To be the most productive coder, it’s important to understand how virtual environments can help product reproducibility and stability of your analysis ([[Python_virtual_environments|Python virtual environments]]) and how to keep running jobs even when you are not connected to the server ([[Terminal_multiplexers|Terminal Multiplexers]]). If you’re less aware than you’d like, we encourage you to visit either of these pages. General Linux commands and getting connected & passing files between you and the server can all be found on our [[Linux_and_department_servers|Linux and the department server page]].

== Submitting jobs via the command line ==

Before saving a code, you can start up python with a <code>ipython</code> or <code>python</code> call to the command line (possibly after activating a virtual environment). This will give you an interactive terminal to test python code out.

Beyond this iterative coding / explorative approach, you may find that writing up complex analysis that requires running a large number of lines of code (without having to manually run) large jobs that you’d like to run on a server, a common approach is to write a <code>.py</code> file to be run all at once, which can be done follows:
<syntaxhighlight lang="bash">ipython filename.py
python filename.py</syntaxhighlight>

Especially when it comes to code run on the server, it’s possible that you’d like to take in a few non-fixed parameters when you run your file. This can be done through the following approach (and is also documented in a tutorial: [https://www.geeksforgeeks.org/how-to-use-sys-argv-in-python/ geeksforgeeks]).

In your python file (<code>filename.py</code>) (often at the top) you deal with arguments passed in from the command line:
<syntaxhighlight lang="python">import sys

input_arguments = sys.argv
arg1 = int(sys.argv[1]) # the first argument (sys.argv[0]) is the filename.
arg2 = sys.argv[2]
...</syntaxhighlight>

If you then ran the following in the command line:
<syntaxhighlight lang="bash">ipython filename.py 1000 happy
python filename.py 1000 happy</syntaxhighlight>

You’d then find the lines below <code>arg2 = sys.argv[2]</code> would have <code>arg1 = 1000</code> (an integer), and <code>arg2 = "happy"</code>. Note that all parameters are pasted through as strings.

=== interactive <code>ipython</code>/<code>python</code> ===

Especially since you’ll be doing more of these command line running of files, if you are not 100% sure your code will run as expected or would like to be able to manipulate some of the objects you’ve made after run your file you can run your scripts interactively. If the script fails (or just when it ends), you’ll obtain an interactive ipython / python output. Below are two examples of how to do so (just use the <code>-i</code> command).
<syntaxhighlight lang="bash">ipython -i filename.py 1001 sad
python -i filename.py 1001 sad</syntaxhighlight>

= Jupyter notebooks via a server =

You can still run jupyter notebooks from a server, which can be useful with large data sets, need for more computing power, or just the desire to not have your personal computer’s cores being taxed. Starting a jupyter notebook from the server is pretty simple, and we outline how to below ([https://coderwall.com/p/ohk6cg/remote-access-to-ipython-notebooks-via-ssh coderwall] is has a good resource).

<ol start="1" style="list-style-type: decimal;">
<li>The first step is to initialize the notebook server just like you would on your own computer. On your remote server do:
<syntaxhighlight lang="bash">remote_server$ jupyter notebook --no-browser --port=8889</syntaxhighlight>

Note that you should think about be running this code after a <code>screen</code> or <code>tmux</code> call so it lasts even if you close / lose connection to the the connection that ran this call.
</li>
<li>Then on your local machine, you can run:</li>
<syntaxhighlight lang="bash">local$ ssh -N -f -L localhost:8888:localhost:8889 user_id@remote_server</syntaxhighlight>
<li>Finally, you can return to the server terminal you launched the jupyter notebook from and you’ll see something like:</li>
<syntaxhighlight lang="bash">Copy/paste this URL into your browser when you connect for the first time,
to login with a token:
http://localhost:8889/?token=13cc54b4a3e13877cb07211346a363f9147a8cf2109263f1&token=13cc54b4a3e13877cb07211346a363f9147a8cf2109263f1
</syntaxhighlight>
<li>Copy this address but substitute <code>8889</code> with <code>8888</code> on your local machine’s web server.</li>
<li>start coding</li>
</ol>

Linux and department servers

2026-07-07T14:22:28Z

Areinhar: formatting tweaks

[[Category:Department Servers]]
[[Category:Remote Server Access]]

The department has two sets of servers, the Leviathans and the Hydras, that can help you run large analyses and gain a better understanding of common command line structure for more high scale computing. One can also view the department servers a bit more as a “sandbox” or “playground” as there are very few constraints and it’s okay to not run as polished code as you might on the [[PSC]] or [[Wright GPU Cluster]].

['''Request:''' All users - if you see comments that are outdated / less helpful than expected please contribute.]

== What resources exist ==

# The department provides 11 leviathan servers (named {<code>leviathan1</code>, ..., <code>leviathan11</code>}<code>.stat.cmu.edu</code>).
# The department provides 3 Ghidorah GPU servers (named {<code>ghidorah1</code>, ..., <code>ghidorah3</code>}<code>.stat.cmu.edu</code>).
# The Ghidorahs and Leviathans all share the same data file system. This allows you to upload a file to any of the servers and then have it on all of them.

''See [[what_resources_when| What resources when]] for guides to help decide which resource you should be using.''

==Introduction to connecting to the server and useful Linux commands==

The department servers all run Linux and require you to interact with them remotely. While you can use a traditional command line, modern IDEs like VS Code also allow for highly integrated graphical interaction.

===Getting Access: (<code>ssh</code>)===

To get onto the server and execute code, you’ll need to access it through <code>ssh</code>. Modern operating systems (including Mac, Linux, and Windows 10/11) have this tool built-in natively.

<ul>
<li>'''Mac / Linux:''' Open up a <code>Terminal</code>.</li>
<li>'''Windows:''' Open up <code>Windows PowerShell</code> or <code>Windows Terminal</code>.</li>
</ul>

In your terminal, type:
<syntaxhighlight lang="bash">ssh andrew_id@leviathan1.stat.cmu.edu</syntaxhighlight>

You will then be prompted to type in your password. (Note: as you type, no characters will appear on the screen for security reasons. Just type it and hit Enter). If you don’t know your password or have never accessed the server before, email [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu].

If you’re accessing the servers frequently, we highly recommend setting up passwordless SSH (see below).

=== Passwordless SSH ===
To connect to the server without typing in your password every time, you need to generate an SSH key pair. This creates a private key that stays securely on your laptop, and a public key that the servers use to recognize you.

* Open up your terminal (Mac/Linux) or PowerShell (Windows).
* Check whether you already have a key pair by looking in your <code>.ssh</code> folder (run <code>ls ~/.ssh</code> on Mac/Linux, or <code>dir ~\.ssh</code> on Windows). If you have files like <code>id_ed25519</code> or <code>id_rsa</code>, you already have a key pair and can skip to the upload step.
* To generate a new key pair, we recommend the Ed25519 algorithm. Run:
<syntaxhighlight lang="bash">ssh-keygen -t ed25519</syntaxhighlight>
* Hit Enter to accept the default file location. When prompted for a passphrase, you can enter one for extra security or leave it empty for true passwordless login.
* Next, upload the public key to the remote server. Because the hydras and ghidorahs share a file system, you only need to do this once.
** '''For Mac/Linux:''' Use the built-in <code>ssh-copy-id</code> command:
** <syntaxhighlight lang="bash">ssh-copy-id andrew_id@leviathan1.stat.cmu.edu</syntaxhighlight>
** '''For Windows:''' Windows OpenSSH does not natively include the <code>ssh-copy-id</code> command. Instead, use this PowerShell one-liner to securely copy your key over:
** <syntaxhighlight lang="powershell">Get-Content "$env:USERPROFILE\.ssh\id_ed25519.pub" | ssh andrew_id@leviathan1.stat.cmu.edu "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys"</syntaxhighlight>
* To ensure the permissions of the <code>~/.ssh</code> directory on the server are strictly enforced, log into the server normally and run:
<syntaxhighlight lang="bash">
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
</syntaxhighlight>
You should now be able to SSH into the server without being prompted for a password.

===File sharing (<code>scp</code> + GUIs)===

There are two primary ways to move files between your personal computer and the server: using the command line (<code>scp</code>) or using a graphical interface (like VS Code or Cyberduck).

====Command line <code>scp</code>====

Because Windows now includes OpenSSH natively, the <code>scp</code> (secure copy) command works exactly the same across Mac, Linux, and Windows PowerShell.

<ol>
<li>Copy from server to your computer
<syntaxhighlight lang="bash">scp andrew_id@leviathan1.stat.cmu.edu:/path/to/remote/file.txt /local/directory/</syntaxhighlight>
To copy an entire folder, add the <code>-r</code> (recursive) flag:
<syntaxhighlight lang="bash">scp -r andrew_id@leviathan1.stat.cmu.edu:/path/to/remote/folder/ /local/directory/</syntaxhighlight>
</li>
<li>Copy from your computer to the server
<syntaxhighlight lang="bash">scp /local/file.txt andrew_id@leviathan1.stat.cmu.edu:/remote/directory/</syntaxhighlight>
</li>
</ol>

====GUI “drag-and-drop” file sharing====
Many students prefer graphical user interfaces (GUIs) to transfer files instead of using the command line. These tools allow you to visually drag and drop files from your computer to the server (and vice-versa).

* '''Visual Studio Code (Recommended for all OS):''' If you use VS Code with the Remote-SSH extension (see "Text Editors" below), you can simply drag and drop files from your desktop directly into the VS Code file explorer pane. This is the easiest and most integrated method.
* '''Mac:''' [https://cyberduck.io/ Cyberduck] remains a solid dedicated tool.
* '''Windows:''' [https://winscp.net/eng/download.php WinSCP] is a popular dedicated GUI, though legacy SSH clients like PuTTY are no longer necessary for modern Windows file transfers.

=== Basic CLI (Linux command line): <code>bash</code>===

Understanding the basic Linux command line is important to navigate when you are on the server. Below are some basic commands to move around the file system and manipulate files.

{| class="wikitable"
|-
!command
!translation
|-
|
<code>pwd</code>
|print the current working directory
|-
|
<code>cd _dir_</code>
| change directory
|-
|
<code>rm _file_</code>
|remove / delete file
|-
|
<code>mv _file_ _dir_</code>
|move file (can change name)
|-
|
<code>cp _file_ _dir_</code>
|copy file
|-
|
<code>ls _dir_</code>
|list files in directory
|}

Use <code>man _command_</code> to get a listing / manual of the command.

Additional resources geared toward statistical computing can be found here: [http://www.jarrodmillman.com/rcsds/standard/bash.html UCB 2015 reproducible DS lecture notes].

'''TODO: add information about <code>sed</code>, <code>chmod</code>, and a link to bash files.'''

==Text editors and IDEs==

Historically, editing code on a server required using terminal-based text editors. While still useful for quick changes, modern tools allow you to use a full graphical IDE on your laptop that computes natively on the server.

=== Visual Studio Code (Highly Recommended) ===
[https://code.visualstudio.com/ Visual Studio Code] (VS Code) is the industry standard for remote development. By using the Microsoft '''Remote - SSH''' extension, you can connect VS Code directly to the department servers.

# Install VS Code on your personal machine.
# Go to the Extensions tab and install '''Remote - SSH'''.
# Click the remote indicator (the green arrows) in the bottom-left corner of VS Code, select "Connect to Host...", and type <code>andrew_id@leviathan1.stat.cmu.edu</code>.
# Once connected, you can open server folders, edit scripts with full syntax highlighting, run interactive terminal sessions, and drag-and-drop files between your laptop and the server all from within the VS Code window.

=== Terminal Editors ===
If you only need to make a quick tweak to a file while already logged into the terminal, you can use built-in editors:
* <code>nano _filename_</code>: A very simple, beginner-friendly text editor. Use <code>Ctrl+O</code> to save and <code>Ctrl+X</code> to exit.
* <code>vim _filename_</code>: A highly powerful but steep-learning-curve text editor.

==Using the servers respectfully==
Because the servers are a shared department resource it is important to use them respectfully. There are many servers and multiple cores (CPUs) per server. To be respectful of other users:

#'''Check the current usage''' of the server you're on before using it.
#Do not use '''all''' the cores on a server, and do not use '''all''' the servers at the same time, so others can get work done.
#If you have a long-running job, '''deprioritize it''' so others can run too.
#Use <code>/scratch</code> for jobs that require intense disk I/O operations

Each of these is easy to check.

=== Checking current usage===
To check which servers are currently being used and which are open, see the [https://info.stat.cmu.edu/grafana/d/beuw1q6y3dgjkc/hydra-cluster?orgId=1 Grafana dashboard for the hydras] and [https://info.stat.cmu.edu/grafana/d/asjkldfbnvajdsfh213/leviathan-cluster?orgId=1 for the Leviathans]. It lists them all and their current usage.

When you're logged in to one node, you can use the <code>top</code> command. <code>top</code> displays the current usage of the server’s resources with a list of jobs run. It also tells you the percentage of the cores that are being used and the amount of RAM (memory) that is available. See [https://www.howtogeek.com/668986/how-to-use-the-linux-top-command-and-understand-its-output/ howtogeek] resources on how to interpret it all.

===Limiting the cores you use===
We recommend that when using parallel processing, use fewer than the total number of cores on the server. Most parallel processing packages let you choose the number of cores to use; the hydras have 12, so pick something smaller.

=== Deprioritize when possible===
<code>nice/renice</code> allows you to lower the priority of your jobs, so the operating system lets other processes run first. See [https://www.howtogeek.com/411979/how-to-set-process-priorities-with-the-nice-and-renice-commands-in-linux/ howtogeek] resources on the basics. <code>nice</code> sets the priority, and <code>renice</code> updates your priority. On the department servers, everyone starts at the same priority, and you can only '''deprioritize''' your jobs -- you can't make them higher priority than everyone else.<syntaxhighlight lang="bash">nice -20 Rscript some-big-script.R
renice -n 20 processid # get the process ID from top</syntaxhighlight>

===Using <code>/scratch</code>===
On department servers, your home directory (the default location you start in when you ssh) is not actually located on the physical server itself. Instead, every department server uses network attached storage (NAS), where everybody's home directly lives on a central storage server and the individual Leviathans and Ghidorahs connect to that storage server. This is the functionality that lets your files follow you no matter what node you log into.

While computing processes on one server do not affect processes on other servers, every server shares the same NAS. Therefore, '''if a user is reading and writing a ton of data to their home directory, it will slow down reading and writing for everyone, even if you aren't on the same server as this user.''' Even if your job does not read or write much data, disk I/O slowdowns will affect common operations, such as ssh, importing packages, and uploading/downloading files.

If you need to run a job that requires a ton of reading and writing data to disk, '''do not read/write using your home directory'''. Instead, use the <code>/scratch</code> folder located on every server. This folder points to an SSD that is specific to that server. Not only will read and writes to <code>/scratch</code> not affect users on other servers, but it will also be generally faster than using your home directory even if nobody else is online. '''If you plan to use''' <code>/scratch</code>''', you should make a new folder inside of''' <code>/scratch</code> '''first and put your data inside that folder. For example:'''<syntaxhighlight lang="bash">
cd /scratch
mkdir my_data # Default permission allows only you to modify contents
cd my_data
# Do stuff in my_data
</syntaxhighlight>It is also very important to remember that <code>/scratch</code> '''is specific to each Leviathan and Ghidorah and will not follow you between servers'''. Thus, you should not use <code>/scratch</code> for long-term storage or for important data. In addition, space on <code>/scratch</code> is limited, so be sure to clean up your data once you don't need it anymore.
==Running jobs==

We have included thoughts about running jobs in two places.

# The first looks at tools that allow you to keep running jobs on the server when you close the connection to the server (turn off your laptop, etc.). These tools are call [[terminal multiplexers]]. 
#The second set of tools focus on how to run your [[R_on_the_server|R analysis]] or [[Python_on_the_server|Python analysis]] code on the server.

==Server specs==

Generally speaking, our servers have multiple CPU cores, but no GPU cores. For deep learning tasks that need GPUs, consider using the [[GPU Cluster|GPU cluster]].

To examine the specs yourself, use
<syntaxhighlight lang="bash">cat /proc/meminfo
cat /proc/cpuinfo</syntaxhighlight>

to see some information.

Building Workflows on Remote Servers

2026-07-07T14:19:51Z

Areinhar: use a grid of boxes instead of a sequence

[[Category:Beginner's_Guide_Scaling|01]]

{{ScalingSidebar}}

Even with perfect parallelization, research experiments can quickly scale beyond the capabilities of any personal computing device. Thus, most workflows with high computing requirements are often performed on High-Performance Computing (HPC) clusters, which are dedicated servers for large computational workloads. These servers often have hardware orders of magnitude faster than personal devices.

Students in the CMU Department of Statistics and Data Science have access to several HPC clusters. Each cluster is configured for various types and sizes of workflows.

Back to [[Beginner's Guide to Computation in Statistical Research]]

== 1. Department Servers ==
The department wholly owns and operates a small HPC cluster, composed of several CPU-only nodes known as the Leviathans, and a few GPU nodes known as the Ghidorahs. The department cluster is suitable for modest sized workflows that would be too slow to run on a personal device.

<div style="display: grid; grid-template-columns: repeat(auto-fit, minmax(500px, 1fr))">
<div style="background-color: #fff3cd; border-left: 5px solid #ffc107; padding: 15px; margin-bottom: 20px; margin-top: 15px; overflow: hidden;">
'''Important: Resource Limits'''<br>
If your workflow requires continual, simultaneous use of multiple Leviathans or Ghidorahs, we strongly recommend migrating to the external shared servers in the next section to ensure fair use for all students.
</div>

<div style="background-color: #f8f9fa; border-left: 4px solid #6c757d; padding: 15px; margin-bottom: 10px;">
'''[[Linux and department servers]]'''<br>
Introduces basic Linux commands and instructions for connecting to the department servers. Start here for all server-based workflows.
</div>

<div style="background-color: #f8f9fa; border-left: 4px solid #75AADB; padding: 15px; margin-bottom: 10px;">
'''[[R on the server]]'''<br>
Demonstrates how to run R scripts and RStudio on the department servers.
</div>

<div style="background-color: #f8f9fa; border-left: 4px solid #005A9C; padding: 15px; margin-bottom: 10px;">
'''[[Python on the server]]'''<br>
Demonstrates how to run Python scripts and Jupyter notebooks on the department servers.
</div>
</div>

== 2. External Shared Servers ==
Students in the department also have access to other HPC resources that are shared between departments and institutions. These clusters are built for massive parallelization and GPU-intensive workloads.

<div style="display: grid; grid-template-columns: repeat(auto-fit, minmax(500px, 1fr))">
<div style="background-color: #f8f9fa; border-left: 4px solid #dc3545; padding: 15px; margin-bottom: 10px;">
'''[[Slurm]]'''<br>
Introduces the job scheduling system used on larger HPC clusters (like Wright and PSC). Essential reading for medium to large-scale server workflows.
</div>

<div style="background-color: #f8f9fa; border-left: 4px solid #28a745; padding: 15px; margin-bottom: 10px;">
'''[[GPU Cluster | Wright GPU Cluster]]'''<br>
The manual for accessing and using the Wright cluster. Ideal for large-scale, GPU-heavy server workflows.
</div>

<div style="background-color: #f8f9fa; border-left: 4px solid #6f42c1; padding: 15px; margin-bottom: 10px;">
'''[[PSC]] (Incomplete)'''<br>
Instructions for accessing and using the Pittsburgh Supercomputing Center (PSC) servers for massive-scale computing.
</div>
</div>

== 3. Other Useful Tools ==
Below are some useful tools and tricks for streamlining your work on any remote server.

<div style="background-color: #f8f9fa; border-left: 4px solid #17a2b8; padding: 15px; margin-bottom: 10px;">
'''[[VS Code Remote]]'''<br>
Introduces the VS Code Remote plugin, which allows you to edit files and run terminals using the full VS Code IDE experience directly on a remote server.
</div>

Beginner's Guide to Computation in Statistical Research

2026-07-07T14:18:01Z

Areinhar: use a grid of boxes instead of a sequence

At some point, you will need to demonstrate that your statistical methods actually work in practice. Whether you are running a toy simulation, analyzing real-world data, or optimizing a complex neural network, you will need to write code. If you are new to computing, the wiki can be overwhelming. '''Read every section top to bottom up through the Reproducibility and Collaboration section. (You do not need to read every page in every section, but note their existence)''' It is designed to take you from installing your first language to running massive workflows on department servers.

== Implementation ==
Your first goal is to translate mathematical algorithms into functional code. '''It is important to familiarize yourself with these concepts early on.'''

<div style="display: grid; grid-template-columns: repeat(auto-fit, minmax(500px, 1fr))">
<div style="background-color: #f8f9fa; border-left: 4px solid #C41230; padding: 10px 15px; margin-bottom: 10px;">
'''[[Choosing and Installing a Programming Language]]''' <br>
Help deciding between R, Python, Julia, or other languages and how to get them running on your personal machine.
</div>

<div style="background-color: #f8f9fa; border-left: 4px solid #C41230; padding: 10px 15px; margin-bottom: 10px;">
'''[[Installing and Configuring an Integrated Development Environment]]''' <br>
Your IDE (Integrated Development Environment) is your workshop. Learn how to set up VS Code or RStudio.
</div>

<div style="background-color: #f8f9fa; border-left: 4px solid #C41230; padding: 10px 15px; margin-bottom: 10px;">
'''[[Beyond Print Statements: Breakpoint Debugging]]''' <br>
Your IDE (Integrated Development Environment) is your workshop. Learn how to set up VS Code or RStudio.
</div>

<div style="background-color: #f8f9fa; border-left: 4px solid #C41230; padding: 10px 15px; margin-bottom: 10px;">
'''[[Command Line Basics]]''' <br>
Essential survival skills for navigating your computer without graphical windows
</div>
</div>
#

== Scale and Efficiency ==
Once you have gotten the hang of implementing statistical algorithms in code, the next major hurdle will be making your workflows '''scalable and efficient'''. For code running on your own machine, you want to make sure you are using every CPU core and every bit of RAM as efficiently as possible. Even then, your personal computer alone can only get you so far. Thankfully, the department also provides several options for remote high-performance computing for even the largest of research workflows.

<div style="display: grid; grid-template-columns: repeat(auto-fit, minmax(500px, 1fr))">
<div style="background-color: #f8f9fa; border-left: 4px solid #C41230; padding: 10px 15px; margin-bottom: 10px;">
'''[[Building Workflows on Remote Servers]]''' <br>
Using the department servers and other computing clusters
</div>

<div style="background-color: #f8f9fa; border-left: 4px solid #C41230; padding: 10px 15px; margin-bottom: 10px;">
'''[[Parallelization: Making the most of your Machine]]''' <br>
How to run multiple experiments simultaneously without babysitting your machine
</div>

<div style="background-color: #f8f9fa; border-left: 4px solid #C41230; padding: 10px 15px; margin-bottom: 10px;">
'''[[Big Data Management]]''' <br>
How to handle datasets that are measured in terabytes
</div>

</div>

== Reproducibility and Collaboration ==
While it is important that you are able to obtain results efficiently, it is also equally important for other members of the academic community to arrive at your same results independently. Specifically, others should be able to run your workflow and obtain the exact same output as you did with minimal tinkering and friction.

<div style="display: grid; grid-template-columns: repeat(auto-fit, minmax(500px, 1fr))">
<div style="background-color: #f8f9fa; border-left: 4px solid #C41230; padding: 10px 15px; margin-bottom: 10px;">
'''[[Version Control]]''' <br>
Keeping track of code/experiment history
</div>

<div style="background-color: #f8f9fa; border-left: 4px solid #C41230; padding: 10px 15px; margin-bottom: 10px;">
'''[[Dependency Management]]''' <br>
Making sure anyone else in the world can run your workflow
</div>

</div>

== Software Development (Optional) ==
If you envision yourself making your methods and associated algorithms available for mass adoption, there are other considerations for publishing your code that go beyond simple reproduciblity. Other should not only be able to reproduce your results, but also modify your code for new experiments with minimal hassle.

<div style="display: grid; grid-template-columns: repeat(auto-fit, minmax(500px, 1fr))">
<div style="background-color: #f8f9fa; border-left: 4px solid #C41230; padding: 10px 15px; margin-bottom: 10px;">
'''[[Creating Libraries and Packages]]''' <br>
How to structure your code to for easy consumption by other researchers
</div>

<div style="background-color: #f8f9fa; border-left: 4px solid #C41230; padding: 10px 15px; margin-bottom: 10px;">
'''[[Working with package repositories]] (TBW)''' <br>
How to upload your packages to public repositories for easy widespread distribution.
</div>

</div>

Main Page

2026-07-07T14:16:14Z

Areinhar: use a grid of boxes instead of a sequence

Welcome to the CMU Department of Statistics and Data Science Computing Wiki! To get the most out of this resource, please note the following:

<div style="background-color: #e2e3e5; border: 5px solid #6c757d; padding: 15px; margin-bottom: 20px; margin-top: 15px;">
'''Purpose & Contribution'''<br>
The main purpose of this wiki is to familiarize readers with the computing resources provided by the department.
* '''Newcomers:''' Please read the [[Beginner's Guide to Computation in Statistical Research]] to familiarize yourself with statistical computing tools and best practices as you begin you research.
* '''Computational Researchers:''' See the [[Article Catalog]] for a full selection of topics and pages and the [[Directory of Computing Resources for Statistical Research]] for available computational resources. You can also use the search bar at the top of this page.
* '''Contributors and Maintainers:''' Feel free to leave a comment on any page, or create your own pages. Please see [[Generic Wiki Information]] for more information.
</div>

== Important Announcements ==
Before using any department computing resources, please see [[Research Computing Dos and Don'ts]]<div style="background-color: #fff8e6; border-left: 4px solid #ffc107; padding: 15px; margin-bottom: 20px;">
* '''[2026-04-10] New Article:''' Introducing [[Globus]], a tool for sharing large files within and between institutions.
* '''[2026-04-01] PSA:''' If you use VS Code, look for "[https://marketplace.visualstudio.com/items?itemName=ms-python.vscode-python-envs Python Environments]" in your extensions and '''uninstall it immediately''' due to environment conflicts.
* '''[2026-04-01] Updated Article:''' The [[Linux and department servers]] page has been updated with instructions on how to use <code>/scratch</code> to avoid disk I/O slowdowns. See [[Linux and department servers#Using /scratch]].
</div>

== Quick Start ==
<div style="display: grid; grid-template-columns: repeat(auto-fit, minmax(500px, 1fr))">
<div style="background-color: #f8f9fa; border-left: 4px solid #005A9C; padding: 15px; margin-bottom: 10px;">
'''[[Beginner's Guide to Computation in Statistical Research]]'''<br>
The essential starting point for understanding how to scale your statistical workflows and choose the right tools for your research.
</div>

<div style="background-color: #f8f9fa; border-left: 4px solid #28a745; padding: 15px; margin-bottom: 10px;">
'''[[Directory of Computing Resources for Statistical Research]]'''<br>
A comprehensive index of all available clusters, servers, and computing environments accessible to department members.
</div>

<div style="background-color: #f8f9fa; border-left: 4px solid #6f42c1; padding: 15px; margin-bottom: 10px;">
'''[[Department Tools and Services]]'''<br>
Additional department tools, printing, web hosting, software licenses, and specialized computing services.
</div>

<div style="background-color: #f8f9fa; border-left: 4px solid #eff07d; padding: 15px; margin-bottom: 10px;">
'''[[Article Catalog]]'''<br>
A comprehensive list of articles organized by topic
</div>

</div>

== Support Resources ==
For assistance with issues not covered in this wiki, please reach out to the following contacts:

<div style="display: grid; grid-template-columns: repeat(auto-fit, minmax(500px, 1fr))">
<div style="background-color: #f8f9fa; border-left: 4px solid #6c757d; padding: 15px; margin-bottom: 10px;">
'''General Tech Support'''<br>
For general issues with department computing resources:<br>
'''Email:''' [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu]
</div>

<div style="background-color: #f8f9fa; border-left: 4px solid #17a2b8; padding: 15px; margin-bottom: 10px;">
'''Research Project Assistance (Graduate Students Only)'''<br>
Live assistance from Alex Shen, PhD Computing TA, for project-specific computing questions:<br>
'''Office Hours:''' Mondays 2:00 PM - 3:00 PM @ FMS Interview Room 320<br>
'''Contact:''' [mailto:ajshen@andrew.cmu.edu ajshen@andrew.cmu.edu] or reach out directly via Slack.
</div>

<div style="background-color: #f8f9fa; border-left: 4px solid #17a2b8; padding: 15px; margin-bottom: 10px;">
'''Onboarding & Troubleshooting (Graduate Students Only)'''<br>
Live assistance for setting up and troubleshooting computing resources:<br>
'''Carl Skipper:''' Office hours by appointment. Request via [mailto:skipper@andrew.cmu.edu skipper@andrew.cmu.edu]<br>
'''Jake Gordon:''' Office hours by appointment. Request via [mailto:jacobg2@andrew.cmu.edu jacobg2@andrew.cmu.edu]
</div>
</div>

== Suggestions Box and Todos ==

<div style="background-color: #aff0ff; border: 4px solid #17a2b8; padding: 15px; margin-bottom: 10px;">
* '''[New Article]''' Beginner's Guide -> Scaling -> Version Control -> Experiment Management
* '''[New Article]''' Beginner's Guide -> Scaling -> Big Data Management -> Articles about file formats and compression
* '''[New Article]''' Beginner's Guide -> Software Development -> Package Creation -> Writing Documentation
* '''[New Article]''' Beginner's Guide -> Software Development -> Package Repositories

</div>

This section contains work in progress and future suggestions for the wiki. If you would like to leave a suggestions, you may:
* Directly contact the Computing TA
* Leave a comment on a specific wiki article with your suggestions
* Leave a comment on this page with your suggestions
<br>

Main Page new

2026-06-29T14:55:37Z

Areinhar:

Welcome to the CMU Department of Statistics and Data Science Computing Wiki! To get the most out of this resource, please note the following:

<div style="background-color: #e2e3e5; border: 5px solid #6c757d; padding: 15px; margin-bottom: 20px; margin-top: 15px;">
'''Purpose & Contribution'''<br>
* The main purpose of this wiki is to familiarize readers with the computing resources '''provided by the department'''. We encourage everyone, especially newcomers, to at least skim all of the articles in the '''Quick Start''' section below.
* This wiki is editable by any current member of the CMU Statistics department. If you find any small errors, feel free to correct them directly. If you have a question or suggestions, please leave a comment on any page in the wiki.
</div>

== Important Announcements ==
Before using any department computing resources, please see [[Research Computing Dos and Don'ts]]<div style="background-color: #fff8e6; border-left: 4px solid #ffc107; padding: 15px; margin-bottom: 20px;">
* '''[2026-04-10] New Article:''' Introducing [[Globus]], a tool for sharing large files within and between institutions.
* '''[2026-04-01] PSA:''' If you use VS Code, look for "[https://marketplace.visualstudio.com/items?itemName=ms-python.vscode-python-envs Python Environments]" in your extensions and '''uninstall it immediately''' due to environment conflicts.
* '''[2026-04-01] Updated Article:''' The [[Linux and department servers]] page has been updated with instructions on how to use <code>/scratch</code> to avoid disk I/O slowdowns. See [[Linux and department servers#Using /scratch]].
</div>

== Quick Start ==
<div style="background-color: #f8f9fa; border-left: 4px solid #005A9C; padding: 15px; margin-bottom: 10px;">
'''[[Beginner's Guide to Computation in Statistical Research]]'''<br>
The essential starting point for understanding how to scale your statistical workflows and choose the right tools for your research.
</div>

<div style="background-color: #f8f9fa; border-left: 4px solid #28a745; padding: 15px; margin-bottom: 10px;">
'''[[Directory of Computing Resources for Statistical Research]]'''<br>
A comprehensive index of all available clusters, servers, and computing environments accessible to department members.
</div>

<div style="background-color: #f8f9fa; border-left: 4px solid #6f42c1; padding: 15px; margin-bottom: 10px;">
'''[[Miscellaneous Services]]'''<br>
Additional department tools, printing, web hosting, software licenses, and specialized computing services.
</div>

<div style="background-color: #f8f9fa; border-left: 4px solid #eff07d; padding: 15px; margin-bottom: 10px;">
'''[[Article Catalog]]'''<br>
A comprehensive list of articles organized by topic
</div>

== Support Resources ==
For assistance with issues not covered in this wiki, please reach out to the following contacts:

<div style="background-color: #f8f9fa; border-left: 4px solid #6c757d; padding: 15px; margin-bottom: 10px;">
'''General Tech Support'''<br>
For general issues with department computing resources:<br>
'''Email:''' [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu]
</div>

<div style="background-color: #f8f9fa; border-left: 4px solid #17a2b8; padding: 15px; margin-bottom: 10px;">
'''Research Project Assistance (Graduate Students Only)'''<br>
Live assistance from Alex Shen, PhD Computing TA, for project-specific computing questions:<br>
'''Office Hours:''' Mondays 2:00 PM - 3:00 PM @ FMS Interview Room 320<br>
'''Contact:''' [mailto:ajshen@andrew.cmu.edu ajshen@andrew.cmu.edu] or reach out directly via Slack.
</div>

<div style="background-color: #f8f9fa; border-left: 4px solid #17a2b8; padding: 15px; margin-bottom: 10px;">
'''Onboarding & Troubleshooting (Graduate Students Only)'''<br>
Live assistance for setting up and troubleshooting computing resources:<br>
'''Carl Skipper:''' Office hours by appointment. Request via [mailto:skipper@andrew.cmu.edu skipper@andrew.cmu.edu]<br>
'''Jake Gordon:''' Office hours by appointment. Request via [mailto:jacobg2@andrew.cmu.edu jacobg2@andrew.cmu.edu]
</div>

== Suggestions Box and Todos ==

<div style="background-color: #aff0ff; border: 4px solid #17a2b8; padding: 15px; margin-bottom: 10px;">
* '''[New Article]''' Beginner's Guide -> Scaling -> Version Control -> Experiment Management
* '''[New Article]''' Beginner's Guide -> Scaling -> Big Data Management -> Articles about file formats and compression
* '''[New Article]''' Beginner's Guide -> Software Development -> Package Creation -> Writing Documentation
* '''[New Article]''' Beginner's Guide -> Software Development -> Package Repositories

</div>

This section contains work in progress and future suggestions for the wiki. If you would like to leave a suggestions, you may:
* Directly contact the Computing TA
* Leave a comment on a specific wiki article with your suggestions
* Leave a comment on this page with your suggestions
<br>

R on the server

2026-06-29T14:49:44Z

Areinhar:

The department's Leviathan cluster has R installed on each machine. It also has RStudio Server running on each machine, so there are two ways to run R code.

== Running R on the server ==

=== Command line ===
On the Leviathans, or on other servers that do not support RStudio, log into the server via SSH first. After getting onto the server (see [[Linux_and_department_servers|Linux and the department server]]), you can start using <code>R</code> but opening an interactive session on the console with

<syntaxhighlight lang="bash">R</syntaxhighlight>

You’ll be able to run code (except for visualization as the server can’t display graphics), from the console.

=== RStudio Server ===
RStudio Server is running on each Leviathan. Choose a Leviathan with low usage. For example, suppose you choose <code>leviathan3</code>. Then type the following URL into your web browser:
<nowiki>http://leviathan3.stat.cmu.edu:8787/</nowiki>
This will open RStudio Server. Use your statistics account (the same one you use to log in via SSH) to log in.

You will be able to use the RStudio files pane to view all your files on the Leviathans. It looks just like RStudio on your own computer. Files are shared across machines, but your RStudio session is specific to the Leviathan you started it on. Return to the same Leviathan to view your running code or output.

== Library Management ==

Compared to your personal computer (and your personal computer using RStudio), installing packages has a few nuances.

<ol>
<li>The first time installing a package, e.g. <syntaxhighlight lang="bash">server$ R

R > install.packages("tidyverse")</syntaxhighlight>
you’ll be asked to set up a local folder to store your files.
</li>
<li>The server’s version of <code>R</code> may different than you personal computer. It will also occasionally update. This will require you to re-install packages (see[[Renv|<code>Renv</code>’s wiki (R package environments)]] for a way to make this less painful).</li>
<li>Compared to RStudio, you may need to debug package installation and dependence structure. Good luck!</li>
<li>Packages that use C, C++, or Fortran code will try to compile it from scratch, instead of downloading a pre-compiled version (like on Windows or macOS). This may require certain libraries to be installed on the servers; if package installation fails and tells you to use <code>apt</code> to install a library, you'll need to ask for help to install these, as you don't have the power to.
</li>
</ol>

== Running Jobs ==

Beyond just opening up an interactive <code>R</code> session with <code lang="bash">R</code>, you can run files in 2 ways (as described below).

=== <code>Rscript</code> ===

<code>Rscript</code> is a more commonly used tool to run <code>.R</code> files, as it allows you to pass additional arguments to the file. For example:

<syntaxhighlight lang="bash">Rscript file.R arg1 arg2</syntaxhighlight>

In order to use <code>arg1</code> and <code>arg2</code> in your <code>R</code> code, you’ll access them (as strings) using <code>commandArgs()</code>, for example:
<pre class="{r}">args <- commandArgs(trailingOnly = TRUE)
if (length(args) > 0) {
first_parameter <- args[1]
second_parameter <- as.numeric(args[2])
...
}</pre>

<code>Rscript</code> also allows for standard output (printed to the terminal while running).

==== Example uses of arguments ====

Examples of argument values abound in statistical analysis. If you're using [[Slurm]] you may use arguments to parallelize a task. These types of tasks could be running through complicated data observations, examining outcomes across a grid of hyperparameters, etc. It could also be useful to pass in the name of a file to process or folder to pull from.

=== <code>R CMD BATCH</code> ===

<code>R CMD BATCH</code> is an older approach (compared to <code>Rscript</code> you cannot pass in additional arguments nor see printouts to the standard output, but a <code>file_name.Rout</code> file will be generated with all print outs of the code.

<syntaxhighlight lang="bash">R CMD BATCH file.R</syntaxhighlight>

If you don't have any input arguments to access and want to save errors and messages that came from running your <code>.R</code> file, this might be a better approach.

== Additional useful tools ==

When approaching running code on different servers (and even on your own computer), it may be useful to understand the amount of time an analysis (or a part of an analysis) will run and how much memory it uses. This can help you decide what computing resources you need / what would make your analysis feasible. Knowing this information can also help you decide if you need to make your code more efficient (which can be done with the help of [http://adv-r.had.co.nz/Profiling.html profiles] and potentially C++ functions - link to come soon).

<ol>
<li>'''Estimating time usage''': To measure the amount of time code takes to run, one approach is to put <pre class="{r}">start <- Sys.time()</pre>

at the start and at the end
<pre class="{r}">used_time <- Sys.time() - start
print(used_time)</pre>If you're running a script, you can also use the <code>time</code> command, as in:<syntaxhighlight lang="shell">
time Rscript some_script.R
</syntaxhighlight></li>
<li>'''Understanding memory usage:''' When you are running code in <code>R</code> or RStudio, you can get some understanding of how large an object is (and therefore get a lower bound on your potential RAM usage) using the package <code>pryr</code>. Calling <code>pryr::object_size(object_name)</code> will tell you what the approximate size of the object is. See [http://adv-r.had.co.nz/memory.html adv-r] on the package usage.

For example,

<syntaxhighlight lang="bash">
R> library(pryr)
R> dm_small <- as.matrix(dist(1:10))
R> object_size(dm_small)
2.1 kB
R> dm_medium <- as.matrix(dist(1:10-))
R> object_size(dm_medium)
87.7 kB
R> dm_large <- as.matrix(dist(1:1000))
R> object_size(dm_large)
8.07 MB
R> dm_large <- as.matrix(dist(1:10000))
R> object_size(dm_large)
801 MB
</syntaxhighlight>

If we did
<syntaxhighlight lang="bash">
R> dm_large <- as.matrix(dist(1:1000000))
</syntaxhighlight>
we'd expect <code>object_size(dm_large)</code> to be around 80 GB, which should crash your personal computer's <code>R</code>/Rstudio since you probably don't have that much RAM.

<br>

It is possible to also check your amount of RAM usage with OS-centric tools (e.g. Mac's: [https://support.apple.com/guide/activity-monitor/view-memory-usage-actmntr1004/mac Activity Monitor] or Window's: [https://www.crucial.com/articles/about-memory/how-to-check-ram-windows-10 Dask Management]).
</li></ol>

[[Category:IDEs]]
[[Category:Department Servers]]

R on the server

2026-06-29T14:49:23Z

Areinhar: hydra -> leviathans

[[Category:IDEs]]
[[Category:Department Servers]]

The department's Leviathan cluster has R installed on each machine. It also has RStudio Server running on each machine, so there are two ways to run R code.

== Running R on the server ==

=== Command line ===
On the Leviathans, or on other servers that do not support RStudio, log into the server via SSH first. After getting onto the server (see [[Linux_and_department_servers|Linux and the department server]]), you can start using <code>R</code> but opening an interactive session on the console with

<syntaxhighlight lang="bash">R</syntaxhighlight>

You’ll be able to run code (except for visualization as the server can’t display graphics), from the console.

=== RStudio Server ===
RStudio Server is running on each Leviathan. Choose a Leviathan with low usage. For example, suppose you choose <code>leviathan3</code>. Then type the following URL into your web browser:
<nowiki>http://leviathan3.stat.cmu.edu:8787/</nowiki>
This will open RStudio Server. Use your statistics account (the same one you use to log in via SSH) to log in.

You will be able to use the RStudio files pane to view all your files on the Leviathans. It looks just like RStudio on your own computer. Files are shared across machines, but your RStudio session is specific to the Leviathan you started it on. Return to the same Leviathan to view your running code or output.

== Library Management ==

Compared to your personal computer (and your personal computer using RStudio), installing packages has a few nuances.

<ol>
<li>The first time installing a package, e.g. <syntaxhighlight lang="bash">server$ R

R > install.packages("tidyverse")</syntaxhighlight>
you’ll be asked to set up a local folder to store your files.
</li>
<li>The server’s version of <code>R</code> may different than you personal computer. It will also occasionally update. This will require you to re-install packages (see[[Renv|<code>Renv</code>’s wiki (R package environments)]] for a way to make this less painful).</li>
<li>Compared to RStudio, you may need to debug package installation and dependence structure. Good luck!</li>
<li>Packages that use C, C++, or Fortran code will try to compile it from scratch, instead of downloading a pre-compiled version (like on Windows or macOS). This may require certain libraries to be installed on the servers; if package installation fails and tells you to use <code>apt</code> to install a library, you'll need to ask for help to install these, as you don't have the power to.
</li>
</ol>

== Running Jobs ==

Beyond just opening up an interactive <code>R</code> session with <code lang="bash">R</code>, you can run files in 2 ways (as described below).

=== <code>Rscript</code> ===

<code>Rscript</code> is a more commonly used tool to run <code>.R</code> files, as it allows you to pass additional arguments to the file. For example:

<syntaxhighlight lang="bash">Rscript file.R arg1 arg2</syntaxhighlight>

In order to use <code>arg1</code> and <code>arg2</code> in your <code>R</code> code, you’ll access them (as strings) using <code>commandArgs()</code>, for example:
<pre class="{r}">args <- commandArgs(trailingOnly = TRUE)
if (length(args) > 0) {
first_parameter <- args[1]
second_parameter <- as.numeric(args[2])
...
}</pre>

<code>Rscript</code> also allows for standard output (printed to the terminal while running).

==== Example uses of arguments ====

Examples of argument values abound in statistical analysis. If you're using [[Slurm]] you may use arguments to parallelize a task. These types of tasks could be running through complicated data observations, examining outcomes across a grid of hyperparameters, etc. It could also be useful to pass in the name of a file to process or folder to pull from.

=== <code>R CMD BATCH</code> ===

<code>R CMD BATCH</code> is an older approach (compared to <code>Rscript</code> you cannot pass in additional arguments nor see printouts to the standard output, but a <code>file_name.Rout</code> file will be generated with all print outs of the code.

<syntaxhighlight lang="bash">R CMD BATCH file.R</syntaxhighlight>

If you don't have any input arguments to access and want to save errors and messages that came from running your <code>.R</code> file, this might be a better approach.

== Additional useful tools ==

When approaching running code on different servers (and even on your own computer), it may be useful to understand the amount of time an analysis (or a part of an analysis) will run and how much memory it uses. This can help you decide what computing resources you need / what would make your analysis feasible. Knowing this information can also help you decide if you need to make your code more efficient (which can be done with the help of [http://adv-r.had.co.nz/Profiling.html profiles] and potentially C++ functions - link to come soon).

<ol>
<li>'''Estimating time usage''': To measure the amount of time code takes to run, one approach is to put <pre class="{r}">start <- Sys.time()</pre>

at the start and at the end
<pre class="{r}">used_time <- Sys.time() - start
print(used_time)</pre>If you're running a script, you can also use the <code>time</code> command, as in:<syntaxhighlight lang="shell">
time Rscript some_script.R
</syntaxhighlight></li>
<li>'''Understanding memory usage:''' When you are running code in <code>R</code> or RStudio, you can get some understanding of how large an object is (and therefore get a lower bound on your potential RAM usage) using the package <code>pryr</code>. Calling <code>pryr::object_size(object_name)</code> will tell you what the approximate size of the object is. See [http://adv-r.had.co.nz/memory.html adv-r] on the package usage.

For example,

<syntaxhighlight lang="bash">
R> library(pryr)
R> dm_small <- as.matrix(dist(1:10))
R> object_size(dm_small)
2.1 kB
R> dm_medium <- as.matrix(dist(1:10-))
R> object_size(dm_medium)
87.7 kB
R> dm_large <- as.matrix(dist(1:1000))
R> object_size(dm_large)
8.07 MB
R> dm_large <- as.matrix(dist(1:10000))
R> object_size(dm_large)
801 MB
</syntaxhighlight>

If we did
<syntaxhighlight lang="bash">
R> dm_large <- as.matrix(dist(1:1000000))
</syntaxhighlight>
we'd expect <code>object_size(dm_large)</code> to be around 80 GB, which should crash your personal computer's <code>R</code>/Rstudio since you probably don't have that much RAM.

<br>

It is possible to also check your amount of RAM usage with OS-centric tools (e.g. Mac's: [https://support.apple.com/guide/activity-monitor/view-memory-usage-actmntr1004/mac Activity Monitor] or Window's: [https://www.crucial.com/articles/about-memory/how-to-check-ram-windows-10 Dask Management]).
</li></ol>

Wright GPU Cluster

2026-04-07T14:35:17Z

Areinhar: /* Job prioritization */ further remove condo mentions, explain prioritization

The department has access to a powerful GPU-based cluster named [https://en.wikipedia.org/wiki/Sewall_Wright Wright] which is hosted here at CMU. This is a 4-node and growing cluster that is a shared resource for Statistics & Data Science, Philosophy, and other departments in Dietrich College. Each department has priority access to its own compute node, but will be able to share the resources from other departments when not in use.

== Access requests ==
This resource is free to use to users in the Departments of Statistics & Data Science, Philosophy, and CMIST. These departments set policies determining which of their users are eligible. Users outside these departments must be sponsored by someone in the eligible departments, as part of a project or collaboration.

To request access, complete the [https://docs.google.com/forms/d/e/1FAIpQLSftCsplE06whGocn62DhwC6NpDSgZ-DMmUacrQID_slIfg6sA/viewform access request form] and our team will be in touch to discuss your request and onboard you.

== Cluster setup ==
The cluster currently consists of 4 servers:
*The head node: You log in to this to write code and run jobs. It has 128 GB of RAM, 16 cores, and '''no GPU'''.
*The compute nodes: Jobs run on these. Each has 256 CPU cores, 8 NVIDIA L40 GPU cards with 48 GB of video RAM each, approximately 1 TB of RAM, and a total of approximately 3 TB of local data/scratch space.
*The file server: Holds 50TB of storage. The head node and compute nodes connect via fiberoptic networking. Your home directory is hosted on the file server, so the same files are accessible on the head node and all compute nodes.
The cluster is currently partitioned. One compute node is reserved for each department involved.

The file server has quotas enabled so that each department gets 1/3rd of the available storage. If a department user tries to add files beyond the quota, the writes will be rejected, and nobody in that department will be able to add files until space is freed. To check the quotas and your own usage, [https://util.stat.cmu.edu/quotareport/ view the quota report].

== Connecting and setting up your environment ==

=== SSH ===

From your terminal, type <code>ssh <YourStatID>@wright.hss.cmu.edu</code> . You can also avoid remembering this by adding the following snippet to your local <code>~/.ssh/config</code> file:
<pre>Host wright
Hostname wright.hss.cmu.edu
User YourStatID
</pre>

You can then connect by simply typing <code>ssh wright</code>.

This connects you to the '''head node.''' Here you can upload your data files, set up your scripts, and test things.

'''Do not run resource-intensive tasks directly on the head node'''. It doesn't have any GPUs, and it has fewer resources than the compute nodes. You can do quick tests to make sure your scripts are set up correctly, but submit all other jobs through Slurm.

=== Uploading files ===
To upload scripts or data files, you can use <code>scp</code> from your local computer (Mac or Linux); on Windows, you can get the Tectia SSH graphical client [https://www.cmu.edu/computing/services/endpoint/software/access.html free through CMU].

You can also connect through the [https://code.visualstudio.com/docs/remote/ssh Remote SSH feature of VS Code] if you like. This is useful to 

*navigate the file system from a proper GUI and 
*modify code/files/folder in place if you are not familiar with the command line
Files uploaded to your home directory will be accessible on the head node and all compute nodes, so they can be freely used in compute jobs.

=== Using Conda ===
The head node has Anaconda installed for managing Python packages and system libraries; see [[Python virtual environments]] for information on how to use it (but note it's '''already installed''', so you don't need to install it from scratch!).

However, the compute nodes '''do not''' have conda, so your slurm jobs can't use <code>conda activate</code> or other familiar commands. Instead, set up your environment on the head node, activate it, install the packages you need, and so on. Then run <code>which python</code> to get the path to the Python executable in your environment. For instance:<syntaxhighlight lang="bash">
$ which python
/home/export/areinhar/.conda/envs/llm/bin/python
</syntaxhighlight>This is the path you'll use in your scripts (see below).

=== Open OnDemand ===

Open OnDemand is a graphical interface where you can run jobs, interactive applications, and manage files. More information can be found [[Open OnDemand Wright|here]].

== Job scheduling and prioritization ==

The GPU cluster manages jobs via [[Slurm]]. Slurm is a job management system that deals with job prioritization and the allocation of resources. On high-performance computers (e.g., PSC and our GPU cluster) Slurm helps allocate resources so multiple jobs can be run quickly. Users run programs by submitting scripts to the Slurm job scheduler, and the scheduler runs them when resources are available.
=== Queue status ===

To check what jobs are running, use <code>squeue</code>. The following options list more helpful information than the default (though you need a wide terminal to read it all):<syntaxhighlight lang="bash">
$ squeue -O "JobID:7,UserName:10,Partition:12,TimeUsed:12,ReasonList,MinCPUS:9,MinMemory:12,tres-per-node,StartTime,EndTime"
JOBID USER PARTITION TIME NODELIST(REASON) MIN_CPUS MIN_MEMORY TRES_PER_NODE START_TIME END_TIME
36229 ebensen all 0:00 (DependencyNeverSati16 128G N/A N/A N/A
36237 ajshen all 12:35:57 n02 2 64G gres:gpu:1 2026-01-26T21:55:23 2026-01-28T21:55:23
36152 msezgin all 1-02:32:04 n03 1 64G gres:gpu:1 2026-01-26T07:59:16 2026-01-28T07:59:16
36132 tsukiant cmist_condo 1-11:00:02 n03 16 8G gres:gpu:1 2026-01-25T23:31:18 2026-01-27T23:31:18
36100 tsukiant cmist_condo 1-17:32:39 n03 16 8G gres:gpu:1 2026-01-25T16:58:41 2026-01-27T16:58:41
36090 tsukiant cmist_condo 1-18:22:51 n03 16 8G gres:gpu:1 2026-01-25T16:08:29 2026-01-27T16:08:29
36247 jwalches statds 42:38 n01 40 15G gres:gpu:3 2026-01-27T09:48:42 2026-01-29T09:48:42
36051 xiangchs statds_condo7:05:24 n01 64 128G gres:gpu:4 2026-01-27T03:25:56 2026-01-29T03:25:56
</syntaxhighlight>

Here we see there are two jobs running on node 1, one on node 2, four on node 3, and one job that is not running because it was submitted with requirements that will not be met. The listed end times are based on the time limit specified when the job was submitted, so the jobs may end sooner. If you submit a job that is waiting for resources, the reason will be listed as <code>(Resources)</code> or <code>(Priority)</code>, and after a few minutes Slurm will list a start time for your job based on when other jobs will finish and free up resources.

To check what resources are currently in use, use <code>sinfo</code>:<syntaxhighlight lang="bash">
$ sinfo -O "NodeList:10,CPUsState:16,Memory:9,FreeMem:10,Gres:20,GresUsed:30"
NODELIST CPUS(A/I/O/T) MEMORY FREE_MEM GRES GRES_USED
n01 64/192/0/256 1000000 221943 gpu:nvidia-l40:8 gpu:nvidia-l40:8(IDX:0-7)
n02 32/224/0/256 1000000 225575 gpu:nvidia-l40:8 gpu:nvidia-l40:0(IDX:N/A)
</syntaxhighlight>This shows that node 1 (Statistics & Data Science) has 64 CPUs allocated and 192 idle, out of 256 total. There is 1TB of RAM, of which 222 GB is free. There are 8 NVIDIA L40 GPus, all 8 of which are in use. Hence if we submit a job to the statds partition requesting any GPUs, it will have to wait until some become free. Meanwhile, node 2 (Philosophy) has no GPUs in use.

=== Partitions ===
The cluster is divided into three main partitions:

* <code>statds</code> (Statistics & Data Science)
* <code>phil</code> (Philosophy)
* <code>cmist</code> (CMIST)

Jobs are submitted to a partition. '''Users must submit jobs to their department's partition''' -- jobs submitted to the wrong partition will be rejected. Partitions correspond to nodes, so this ensures each department's jobs use their own node.

If you want to use other nodes, you can submit jobs to the <code>all</code> partition. This requests any available node from any partition, so your job will run on whichever node has resources available. However, your job will have lower priority, as described below; if your all job runs on the philosophy node, a user who submits to <code>phil</code> can preempt your job, causing it to be killed to make room for them, then restarted after their job is done.

If you want to use a specific node within <code>all</code> you can specify the node within your submit script as shown below.<syntaxhighlight lang="batch">
#SBATCH --nodelist=n03
</syntaxhighlight>

=== Job prioritization ===
Slurm runs jobs based on the resources they request and what is available. If resources aren't available for your job, slurm will put it in the queue and wait until other jobs finish. However, it's not simply a first-in first-out queue -- Slurm attempts to arrange the jobs so every job can run, even if that means running them out of order. Partitions have different quality-of-service (QOS) values that affect their prioritization, and Slurm also adjusts each user's priority based on how many jobs they submit, so that one user can't hog the cluster by submitting dozens of jobs; if one user has many jobs waiting, theirs will be reduced in priority so other users can run jobs as well.

Examples:

* You submit Job A to the <code>all</code> partition. Someone else submits Job B to <code>statds</code>. The <code>all</code> partition has lower QOS, so Job B will get to use the StatDS node first, and Job A will either wait or use a different node.
* You submit Job A to the <code>all</code> partition. It begins to run on the philosophy node. Someone else submits Job B to <code>phil</code>, and there are insufficient resources to run both at the same time. Job A is killed, Job B runs, and Job A is placed back into the queue to be run when resources are available.
* Someone submits Job A that requests 6 GPUs. Later, you submit Job B that requests 4 GPUs on the same partition. The compute node only has 8 GPUs, so Job B will be queued until Job A completes.
* Someone submits Job A that requests 6 GPUs. Later, you submit Job B that requests 2 GPUs. Job B can run immediately.
* You submit a job that requires all 8 GPUs on a node, but 4 are currently in use. Your job is put in the queue and scheduled to run when Slurm knows the current jobs will finish (because of their maximum time limit). Meanwhile, someone submits a job using 2 GPUs and a short time limit. Slurm lets it run immediately, because it knows it will be done before your job is scheduled to run. Other jobs may be put off until after yours runs, so yours can get the 8 GPUs it needs.
* You submit 50 jobs that each require 1 GPU. The node has only 8 GPUs, but several are in use. Several of your jobs run, but the others are queued for later. The fair-sharing system assigns them a lower priority, because you have run so many jobs, and jobs submitted by other users will run first.
You can view the currently running or queued jobs by running <code>squeue</code> on the head node, as described above. The <code>NODELIST(REASON)</code> column will indicate which jobs are queued by giving special codes:

* <code>(Resources)</code> means there are not enough resources (CPUs, memory, or GPUs) available for the job. It will run once the resources are free.
* <code>(Priority)</code> means resources are available, but Slurm wants to run a higher-priority job first. For example, maybe a large job has been waiting for resources for two days before you submit your job; Slurm may make your job wait so the large job can start first.

=== Job etiquette ===
The cluster is shared between users and departments. Therefore:

* Request the minimum resources required for your job, so other users can access other resources.
* If you need to submit a large number of jobs and aren't in a hurry, consider using the <code>all</code> partition. Using <code>all</code> lets you get all available resources, but at lower priority, so other users can run jobs as well and you don't hog the cluster.
* Watch your disk space usage. With large deep models it is easy to use many terabytes of storage, and each department has a quota. [https://util.stat.cmu.edu/quotareport/ You can check the quota and your own usage here.]

== Setting up jobs ==

=== Submitting batch jobs ===
The best way to run jobs is by writing a shell script specifying the Slurm options. To launch a batch job from a script you will need two files:

*the actual script in your favorite programming language that does whatever computation you want
*a bash script that requests the resources you need from Slurm and gives the commands to launch your job.
A typical bash script might look like this:<pre>#!/bin/bash
# specify partition to run in (statds, phil, or cmist)
#SBATCH --partition statds
#SBATCH --ntasks 1
#SBATCH --job-name=nameofyourjob

# number of CPUs you need:
#SBATCH --cpus-per-task 1

# amount of memory per CPU
# (total allocated is CPUs * memory)
#SBATCH --mem-per-cpu 4G

# number of GPUs you need
#SBATCH --gres=gpu:1

# maximum time limit for job, HH:MM:SS (up to 48 hours max, 48:00:00)
#SBATCH --time=01:00:00

# email you when job is finished or errors out:
#SBATCH --mail-user=YourAndrewID@andrew.cmu.edu
#SBATCH --mail-type=END,FAIL

## Choose one:

# to run a Python script in a Conda environment called ENV
srun /home/export/YOURANDREWID/.conda/envs/ENV/bin/python YourScript.py

# to run an R script:
srun Rscript YourScript.R
</pre>Adjust the number of CPUs, number of GPUs, memory usage, and time limit based on the needs of your script. Requesting more than you need may result in your job having to wait for resources to free up; requesting a long time you don't need will block slurm from scheduling other jobs for when yours will be done. (And if you request more resources than exist on the compute nodes, your job will be queued forever while it waits for resources to appear.)

'''Note:''' Slurm batch jobs are limited to a maximum duration of 48 hours. Jobs running longer will be killed.

To run the job, SSH into the head node and navigate to the directory with your scripts. Then run:<syntaxhighlight lang="bash">
sbatch example.sh
</syntaxhighlight>where <code>example.sh</code> is the name of your bash script.

You will see a message like "Submitted batch job 1209" when your job is submitted. Any printed output from your script will automatically be redirected to a file based on the job number -- in this case, <code>slurm-1209.out</code>. You can use <code>tail -f slurm-1209.out</code> to view the output live and monitor your script's progress.

=== Selecting the right resources ===
In the Slurm script above, you can choose the number of tasks, the number of GPUs, the memory per CPU, the number of GPUs, and the time limit. Choose these values wisely:

* You generally want '''1 task''' (<code>--ntasks 1</code>). Asking for multiple tasks will cause srun to run multiple copies of your script simultaneously. If you need to run the same script with 10 different parameter values, this can be useful: request 10 tasks, and read the <code>SLURM_PROCID</code> environment variable in your script. It will be numbered 0-9 (for 10 tasks), so you can use this to set the parameter values for each parallel run.
* Set the number of CPUs based on your script. If your script is entirely single-threaded, ask for 1. If you use parallel processing (such as with multiprocessing in Python or the parallel package in R), use a multithreaded package, or spawn many parallel programs in your job, request more.
* Request the minimum amount of memory (RAM) needed for your script. This is per-CPU, so requesting a lot of CPUs and RAM can easily mean you request more RAM than exists on the node.
* Start with 1 GPU, adding more if your job requires more GPU memory (VRAM) than is on one GPU, or if you explicitly write your script to use more. If you do not use GPUs, request 0 so you aren't allocated a GPU that nobody else can use while your job runs.
* Set the time limit a little longer than you expect your script to run. Slurm uses the time limit when scheduling, so always choosing the maximum (48 hours) will backfire: Slurm will assume your job will take a long time and will run other things first. If you know your job can only take an hour, setting, say, a 01:30:00 time limit will let Slurm fit your job in before other long jobs in the queue. But of course if you choose too low and your job exceeds the limit, it will be killed, so leave yourself some extra time.

=== Interactive mode ===

You can also run interactive sessions that allow you to, say, run an R or Python console and type commands and immediately view the output. From the head node, run a command like:<syntaxhighlight lang="bash">
# remember to edit to correct partition:
srun --partition statds --ntasks 1 \
--cpus-per-task 16 --mem-per-cpu 4G \
--gres=gpu:1 --time=01:00:00 --pty \
bash
</syntaxhighlight>This runs a job that opens a bash shell on the compute node, and connects you directly to this job. You can now run bash commands, open Python, or run R scripts. Note your interactive session has a time limit and will be killed when it elapses, so set the time limit carefully and choose the resource amounts in the command according to your needs.

Interactive sessions are limited to '''two hours maximum'''. You cannot request a longer time limit. This is because interactive sessions are inefficient, with the cluster spending most of its time waiting for you to type something instead of running code. For large tasks, submit a batch job.

=== Interactive via VS Code ===
VS Code supports SSH and tunneling, which allows it to connect to a server and run commands. This allows you to use the Jupyter Notebook support built into VS Code to interact with the GPU cluster. As this uses an interactive session, it's subject to the same '''two-hour time limit'''. Hence notebooks should be used only for exploration and debugging, not large jobs.

To use this, we need to connect to the head node, run an interactive Slurm session, and then connect VS Code to that Slurm session so it can use it. This takes a few extra steps.

==== Setup ====
Before you use this for the first time, do the following steps:

*SSH into Wright and run the following to install and unpack the VSCode CLI: 
<pre>curl -Lk 'https://code.visualstudio.com/sha/download?build=stable&os=cli-alpine-x64' --output vscode_cli.tar.gz
tar -xf vscode_cli.tar.gz
</pre>

*Add the <code>code</code> executable to your PATH by adding the following to your <code>.bashrc</code> file: <code>export PATH="~:$PATH"</code><pre>echo 'export PATH="$PATH:/path/to/code"' >> ~/.bashrc
</pre>
*In VSCode, install the <code>Remote - Tunnels</code> extension, then use the VSCode SSH feature to connect to Wright and make sure to install all the local VS Code extensions on the remote machine as well (e.g., Python, Jupyter, ...)

==== Running a session ====
*In the Wright headnode, launch a Slurm interactive session with <code>srun</code>, following the instructions above.

*Inside the interactive session, run <code>code tunnel</code> and follow the instructions. The first time the system will ask you to name the tunnel and authorize the process via an authenticator, I suggest using GitHub since it usually works quite smoothly. Subsequent tunnels might still ask you to authorize their creation by typing an alphanumerical sequence into a link that appears in the terminal. Once you’ve done that, the tunnel is successfully created.
*From VSCode, select <code>Connect to Tunnel</code>, then choose the authentication method you previously used and you will see the tunnel appear in the dropdown. Select it and you will land directly into the interactive session we just created
*You will now be able to run code and, e.g., Jupyter Notebooks using the compute node allocated by the Slurm interactive session. VS Code will be disconnected when the Slurm session ends, such as when it reaches its time limit.
Note that if you need to use a conda environment, you'll have to configure VS Code to use the Python executable you found above, since conda is not installed on the compute nodes.

=== SSH to Compute Nodes ===
The ability to SSH directly to compute nodes is disabled. You can run the command below to start a bash terminal on a compute node when running a batch job. This allows for the same functionality as SSH.<syntaxhighlight lang="bash">
srun --jobid=yourjobidhere --pty /bin/bash
</syntaxhighlight>

== Monitoring cluster usage ==
The [https://util.stat.cmu.edu/grafana/d/Oxed_c6Wz/nvidia-wright-gpu-metrics NVIDIA Wright GPU Metrics Dashboard] shows graphs of GPU usage across the cluster. At the top left, you can select which node to graph, so viewing your department's node will let you see how GPUs are being used.

If you're developing a long-running job, we recommend checking this dashboard while your job is running. It is easy to accidentally write PyTorch or Tensorflow code that '''doesn't actually use the GPU''', so check the graphs to see if the expected number of GPUs are in used based on your job's requirements and the other jobs listed in <code>squeue</code>. If they are not, get a terminal on the compute node as described above and run <code>nvidia-smi</code> to get GPU usage information, including which scripts are accessing the GPU. If your script doesn't show up, you can debug it to figure out why no GPUs are being used.

== Using scratch storage ==
Each compute node (but not the head node) has several terabytes of scratch space. This is fast SSD storage that is directly attached to the node, rather than being accessed over the network (like all files in <code>/home</code> are). This is useful if your jobs need to write large temporary files: reading and writing the scratch space is faster than reading and writing files in <code>/home</code>.

Your home directory contains a symlink to the scratch space at <code>~/scratch</code>. To use it, create a directory inside named after your username:
mkdir ~/scratch/yourusername
Then you can read and write files in that directory.

'''Warnings:'''

* The scratch space is meant for '''temporary''' storage during a job. It may be cleared out during cluster maintenance or if the disk is full.
* The scratch drives are much smaller than the huge networked disk used for <code>/home</code>, so be careful what you store in it, and regularly delete files you don't need.
* Each node has separate scratch disks, so data you write on one node is '''not''' available from any other node, or from the head node.

The scratch space is thus not for final model outputs or results. It is ideal for things like saving checkpoints while your job is running, dumping data to debug a job, saving output from one script to be immediately used by another script, etc.

Scratch Storage

2026-04-02T14:40:07Z

Areinhar: expand with use cases

Scratch storage is a location where you can store data directly on the machine. These locations use faster flash based storage, which is better suited for the heavy reading and writing of data.

Every time you read/write from your <code>/home/yourusername</code> directory, you are accessing a remote storage location over the network. This is slower to begin with, especially if other users are running read/write heavy jobs.

== Storage Terminology ==
NVME SSD - Fast flash based storage. Smaller available storage size.

Spinning Disk Storage - Slow mechanical hard drive with spinning disks. Larger available storage size

Each Leviathan is equipped with a 4TB NVME SSD (faster than the NAS which uses spinning disks.). This allows for heavy reading and writing at high speeds. Utilizing this space can speed up your jobs and help with congestion on the NAS.

== Use Cases ==
Scratch storage is useful for temporarily storing large files you will be reading and writing from heavily. Because the scratch storage is separate between each machine, not shared, and because it is limited in size, it is less useful for long-term storage.

Good uses include:

* Saving checkpoints or output from long-running jobs or simulations, to be analyzed and moved to long-term storage once the job is done
* Saving output from one script that will immediately be used by another script
* Writing huge outputs from scripts when you're still debugging them and won't need to keep the output for long

Of course, anything saved to <code>/scratch</code> can be moved (with <code>cp</code> or <code>mv</code>) to your home directory if you decide to keep it permanently.

== Scratch Storage Location on Each Leviathan ==
Scratch storage is available in <code>/scratch</code>.

You can view the amount of space used at <code>/scratch</code> for each leviathan [https://info.stat.cmu.edu/grafana/d/asjkldfbnvajdsfh213/leviathan-cluster?orgId=1 here].

Every <code>/scratch</code> location on each leviathan is it's own storage location. Meaning if you create data on Leviathan1 on <code>/scratch</code>, it will not be accessible on Leviathan7 at <code>/scratch</code>.

====== Steps to use Scratch Storage ======

# Make a folder in /scratch location.
#* <code>mkdir /scratch/yourusername</code>
# Change ownership of the folder to you.
#* <code>chown yourusername:yourusername /scratch/yourusername</code>
# Update permissions on folder so only you can read it (modify as needed.)
#* <code>chmod 750 /scratch/yourusername</code>

[[File:ScratchSpace.png|center|frame]]

Wright GPU Cluster

2026-03-30T18:59:06Z

Areinhar: /* Job scheduling and prioritization */ fix link to Slurm

The department has access to a powerful GPU-based cluster named [https://en.wikipedia.org/wiki/Sewall_Wright Wright] which is hosted here at CMU. This is a 4-node and growing cluster that is a shared resource for Statistics & Data Science, Philosophy, and other departments in Dietrich College. Each department has priority access to its own compute node, but will be able to share the resources from other departments when not in use.

== Access requests ==
This resource is free to use to users in the Departments of Statistics & Data Science, Philosophy, and CMIST. These departments set policies determining which of their users are eligible. Users outside these departments must be sponsored by someone in the eligible departments, as part of a project or collaboration.

To request access, complete the [https://docs.google.com/forms/d/e/1FAIpQLSftCsplE06whGocn62DhwC6NpDSgZ-DMmUacrQID_slIfg6sA/viewform access request form] and our team will be in touch to discuss your request and onboard you.

== Cluster setup ==
The cluster currently consists of 4 servers:
*The head node: You log in to this to write code and run jobs. It has 128 GB of RAM, 16 cores, and '''no GPU'''.
*The compute nodes: Jobs run on these. Each has 256 CPU cores, 8 NVIDIA L40 GPU cards with 48 GB of video RAM each, approximately 1 TB of RAM, and a total of approximately 3 TB of local data/scratch space.
*The file server: Holds 50TB of storage. The head node and compute nodes connect via fiberoptic networking. Your home directory is hosted on the file server, so the same files are accessible on the head node and all compute nodes.
The cluster is currently partitioned. One compute node is reserved for each department involved.

The file server has quotas enabled so that each department gets 1/3rd of the available storage. If a department user tries to add files beyond the quota, the writes will be rejected, and nobody in that department will be able to add files until space is freed. To check the quotas and your own usage, [https://util.stat.cmu.edu/quotareport/ view the quota report].

== Connecting and setting up your environment ==

=== SSH ===

From your terminal, type <code>ssh <YourStatID>@wright.hss.cmu.edu</code> . You can also avoid remembering this by adding the following snippet to your local <code>~/.ssh/config</code> file:
<pre>Host wright
Hostname wright.hss.cmu.edu
User YourStatID
</pre>

You can then connect by simply typing <code>ssh wright</code>.

This connects you to the '''head node.''' Here you can upload your data files, set up your scripts, and test things.

'''Do not run resource-intensive tasks directly on the head node'''. It doesn't have any GPUs, and it has fewer resources than the compute nodes. You can do quick tests to make sure your scripts are set up correctly, but submit all other jobs through Slurm.

=== Uploading files ===
To upload scripts or data files, you can use <code>scp</code> from your local computer (Mac or Linux); on Windows, you can get the Tectia SSH graphical client [https://www.cmu.edu/computing/services/endpoint/software/access.html free through CMU].

You can also connect through the [https://code.visualstudio.com/docs/remote/ssh Remote SSH feature of VS Code] if you like. This is useful to 

*navigate the file system from a proper GUI and 
*modify code/files/folder in place if you are not familiar with the command line
Files uploaded to your home directory will be accessible on the head node and all compute nodes, so they can be freely used in compute jobs.

=== Using Conda ===
The head node has Anaconda installed for managing Python packages and system libraries; see [[Python virtual environments]] for information on how to use it (but note it's '''already installed''', so you don't need to install it from scratch!).

However, the compute nodes '''do not''' have conda, so your slurm jobs can't use <code>conda activate</code> or other familiar commands. Instead, set up your environment on the head node, activate it, install the packages you need, and so on. Then run <code>which python</code> to get the path to the Python executable in your environment. For instance:<syntaxhighlight lang="bash">
$ which python
/home/export/areinhar/.conda/envs/llm/bin/python
</syntaxhighlight>This is the path you'll use in your scripts (see below).

=== Open OnDemand ===

Open OnDemand is a graphical interface where you can run jobs, interactive applications, and manage files. More information can be found [[Open OnDemand Wright|here]].

== Job scheduling and prioritization ==

The GPU cluster manages jobs via [[Slurm]]. Slurm is a job management system that deals with job prioritization and the allocation of resources. On high-performance computers (e.g., PSC and our GPU cluster) Slurm helps allocate resources so multiple jobs can be run quickly. Users run programs by submitting scripts to the Slurm job scheduler, and the scheduler runs them when resources are available.
=== Queue status ===

To check what jobs are running, use <code>squeue</code>. The following options list more helpful information than the default (though you need a wide terminal to read it all):<syntaxhighlight lang="bash">
$ squeue -O "JobID:7,UserName:10,Partition:12,TimeUsed:12,ReasonList,MinCPUS:9,MinMemory:12,tres-per-node,StartTime,EndTime"
JOBID USER PARTITION TIME NODELIST(REASON) MIN_CPUS MIN_MEMORY TRES_PER_NODE START_TIME END_TIME
36229 ebensen all 0:00 (DependencyNeverSati16 128G N/A N/A N/A
36237 ajshen all 12:35:57 n02 2 64G gres:gpu:1 2026-01-26T21:55:23 2026-01-28T21:55:23
36152 msezgin all 1-02:32:04 n03 1 64G gres:gpu:1 2026-01-26T07:59:16 2026-01-28T07:59:16
36132 tsukiant cmist_condo 1-11:00:02 n03 16 8G gres:gpu:1 2026-01-25T23:31:18 2026-01-27T23:31:18
36100 tsukiant cmist_condo 1-17:32:39 n03 16 8G gres:gpu:1 2026-01-25T16:58:41 2026-01-27T16:58:41
36090 tsukiant cmist_condo 1-18:22:51 n03 16 8G gres:gpu:1 2026-01-25T16:08:29 2026-01-27T16:08:29
36247 jwalches statds 42:38 n01 40 15G gres:gpu:3 2026-01-27T09:48:42 2026-01-29T09:48:42
36051 xiangchs statds_condo7:05:24 n01 64 128G gres:gpu:4 2026-01-27T03:25:56 2026-01-29T03:25:56
</syntaxhighlight>

Here we see there are two jobs running on node 1, one on node 2, four on node 3, and one job that is not running because it was submitted with requirements that will not be met. The listed end times are based on the time limit specified when the job was submitted, so the jobs may end sooner. If you submit a job that is waiting for resources, the reason will be listed as <code>(Resources)</code> or <code>(Priority)</code>, and after a few minutes Slurm will list a start time for your job based on when other jobs will finish and free up resources.

To check what resources are currently in use, use <code>sinfo</code>:<syntaxhighlight lang="bash">
$ sinfo -O "NodeList:10,CPUsState:16,Memory:9,FreeMem:10,Gres:20,GresUsed:30"
NODELIST CPUS(A/I/O/T) MEMORY FREE_MEM GRES GRES_USED
n01 64/192/0/256 1000000 221943 gpu:nvidia-l40:8 gpu:nvidia-l40:8(IDX:0-7)
n02 32/224/0/256 1000000 225575 gpu:nvidia-l40:8 gpu:nvidia-l40:0(IDX:N/A)
</syntaxhighlight>This shows that node 1 (Statistics & Data Science) has 64 CPUs allocated and 192 idle, out of 256 total. There is 1TB of RAM, of which 222 GB is free. There are 8 NVIDIA L40 GPus, all 8 of which are in use. Hence if we submit a job to the statds partition requesting any GPUs, it will have to wait until some become free. Meanwhile, node 2 (Philosophy) has no GPUs in use.

=== Partitions and condos ===
The cluster is divided into three partitions:

* <code>statds</code> (Statistics & Data Science)
* <code>phil</code> (Philosophy)
* <code>cmist</code> (CMIST)

Jobs are submitted to a partition. '''Users must submit jobs to their department's partition''' -- jobs submitted to the wrong partition will be rejected.

There are several other special partitions:

* <code>all</code> requests any available node from any partition. Thus a Philosophy user can run a job on the CMIST node, or vice versa.
* <code>statds_condo</code>, <code>phil_condo</code>, and <code>cmist_condo</code> are "condo" partitions. Any user can submit to any of these partitions.

These special partitions are like holiday condos: any user can access them, but '''with a lower priority''' than the owner (the host department). Jobs running on them are also preemptible. For example, if a Philosophy user submits to <code>statds_condo</code>, their job will run only when Statistics users are not using the resources. If a Statistics user submits a large job after yours, your job might be terminated, and will automatically run again when resources become available. The <code>all</code> partition is similar, but for whichever node happens to be free.

=== Job prioritization ===
Slurm runs jobs based on the resources they request and what is available. If resources aren't available for your job, slurm will put it in the queue and wait until other jobs finish. However, it's not simply a first-in first-out queue -- Slurm attempts to arrange the jobs so every job can run, even if that means running them out of order. Slurm also adjusts each user's priority based on how many jobs they submit, so that one user can't hog the cluster by submitting dozens of jobs; if one user has many jobs waiting, theirs will be reduced in priority so other users can run jobs as well.

Examples:

* Someone submits Job A that requests 6 GPUs. Later, you submit Job B that requests 4 GPUs on the same partition. The compute node only has 8 GPUs, so Job B will be queued until Job A completes.
* Someone submits Job A that requests 6 GPUs. Later, you submit Job B that requests 2 GPUs. Job B can run immediately.
* You submit a job that requires all 8 GPUs on a node, but 4 are currently in use. Your job is put in the queue and scheduled to run when Slurm knows the current jobs will finish (because of their maximum time limit). Meanwhile, someone submits a job using 2 GPUs and a short time limit. Slurm lets it run immediately, because it knows it will be done before your job is scheduled to run. Other jobs may be put off until after yours runs, so yours can get the 8 GPUs it needs.
* You submit 50 jobs that each require 1 GPU. The node has only 8 GPUs, but several are in use. Several of your jobs run, but the others are queued for later. Other users can submit jobs while yours are in the queue, and some of their jobs may run before some of yours, for fairness.
* You submit a job to a condo partition. The partition is currently full, so your job is queued. Other users from that department submit new jobs. As they are higher priority than you, their jobs run first.
You can view the currently running or queued jobs by running <code>squeue</code> on the head node, as described above. The <code>NODELIST(REASON)</code> column will indicate which jobs are queued by giving special codes:

* <code>(Resources)</code> means there are not enough resources (CPUs, memory, or GPUs) available for the job. It will run once the resources are free.
* <code>(Priority)</code> means resources are available, but Slurm wants to run a higher-priority job first. For example, maybe a large job has been waiting for resources for two days before you submit your job; Slurm may make your job wait so the large job can start first.

=== Job etiquette ===
The cluster is shared between users and departments. Therefore:

* Request the minimum resources required for your job, so other users can access other resources.
* If you need to submit a large number of jobs and aren't in a hurry, consider using your department's condo partition or <code>all</code>. Using <code>all</code> lets you get all available resources, but at lower priority, so other users can run jobs as well and you don't hog the cluster.
* Watch your disk space usage. With large deep models it is easy to use many terabytes of storage, and each department has a quota. [https://util.stat.cmu.edu/quotareport/ You can check the quota and your own usage here.]

== Setting up jobs ==

=== Submitting batch jobs ===
The best way to run jobs is by writing a shell script specifying the Slurm options. To launch a batch job from a script you will need two files:

*the actual script in your favorite programming language that does whatever computation you want
*a bash script that requests the resources you need from Slurm and gives the commands to launch your job.
A typical bash script might look like this:<pre>#!/bin/bash
# specify partition to run in (statds, phil, or cmist)
#SBATCH --partition statds
#SBATCH --ntasks 1
#SBATCH --job-name=nameofyourjob

# number of CPUs you need:
#SBATCH --cpus-per-task 1

# amount of memory per CPU
# (total allocated is CPUs * memory)
#SBATCH --mem-per-cpu 4G

# number of GPUs you need
#SBATCH --gres=gpu:1

# maximum time limit for job, HH:MM:SS (up to 48 hours max, 48:00:00)
#SBATCH --time=01:00:00

# email you when job is finished or errors out:
#SBATCH --mail-user=YourAndrewID@andrew.cmu.edu
#SBATCH --mail-type=END,FAIL

## Choose one:

# to run a Python script in a Conda environment called ENV
srun /home/export/YOURANDREWID/.conda/envs/ENV/bin/python YourScript.py

# to run an R script:
srun Rscript YourScript.R
</pre>Adjust the number of CPUs, number of GPUs, memory usage, and time limit based on the needs of your script. Requesting more than you need may result in your job having to wait for resources to free up; requesting a long time you don't need will block slurm from scheduling other jobs for when yours will be done. (And if you request more resources than exist on the compute nodes, your job will be queued forever while it waits for resources to appear.)

'''Note:''' Slurm batch jobs are limited to a maximum duration of 48 hours. Jobs running longer will be killed.

To run the job, SSH into the head node and navigate to the directory with your scripts. Then run:<syntaxhighlight lang="bash">
sbatch example.sh
</syntaxhighlight>where <code>example.sh</code> is the name of your bash script.

You will see a message like "Submitted batch job 1209" when your job is submitted. Any printed output from your script will automatically be redirected to a file based on the job number -- in this case, <code>slurm-1209.out</code>. You can use <code>tail -f slurm-1209.out</code> to view the output live and monitor your script's progress.

=== Selecting the right resources ===
In the Slurm script above, you can choose the number of tasks, the number of GPUs, the memory per CPU, the number of GPUs, and the time limit. Choose these values wisely:

* You generally want '''1 task''' (<code>--ntasks 1</code>). Asking for multiple tasks will cause srun to run multiple copies of your script simultaneously. If you need to run the same script with 10 different parameter values, this can be useful: request 10 tasks, and read the <code>SLURM_PROCID</code> environment variable in your script. It will be numbered 0-9 (for 10 tasks), so you can use this to set the parameter values for each parallel run.
* Set the number of CPUs based on your script. If your script is entirely single-threaded, ask for 1. If you use parallel processing (such as with multiprocessing in Python or the parallel package in R), use a multithreaded package, or spawn many parallel programs in your job, request more.
* Request the minimum amount of memory (RAM) needed for your script. This is per-CPU, so requesting a lot of CPUs and RAM can easily mean you request more RAM than exists on the node.
* Start with 1 GPU, adding more if your job requires more GPU memory (VRAM) than is on one GPU, or if you explicitly write your script to use more. If you do not use GPUs, request 0 so you aren't allocated a GPU that nobody else can use while your job runs.
* Set the time limit a little longer than you expect your script to run. Slurm uses the time limit when scheduling, so always choosing the maximum (48 hours) will backfire: Slurm will assume your job will take a long time and will run other things first. If you know your job can only take an hour, setting, say, a 01:30:00 time limit will let Slurm fit your job in before other long jobs in the queue. But of course if you choose too low and your job exceeds the limit, it will be killed, so leave yourself some extra time.

=== Interactive mode ===

You can also run interactive sessions that allow you to, say, run an R or Python console and type commands and immediately view the output. From the head node, run a command like:<syntaxhighlight lang="bash">
# remember to edit to correct partition:
srun --partition statds --ntasks 1 \
--cpus-per-task 16 --mem-per-cpu 4G \
--gres=gpu:1 --time=01:00:00 --pty \
bash
</syntaxhighlight>This runs a job that opens a bash shell on the compute node, and connects you directly to this job. You can now run bash commands, open Python, or run R scripts. Note your interactive session has a time limit and will be killed when it elapses, so set the time limit carefully and choose the resource amounts in the command according to your needs.

Interactive sessions are limited to '''two hours maximum'''. You cannot request a longer time limit. This is because interactive sessions are inefficient, with the cluster spending most of its time waiting for you to type something instead of running code. For large tasks, submit a batch job.

=== Interactive via VS Code ===
VS Code supports SSH and tunneling, which allows it to connect to a server and run commands. This allows you to use the Jupyter Notebook support built into VS Code to interact with the GPU cluster. As this uses an interactive session, it's subject to the same '''two-hour time limit'''. Hence notebooks should be used only for exploration and debugging, not large jobs.

To use this, we need to connect to the head node, run an interactive Slurm session, and then connect VS Code to that Slurm session so it can use it. This takes a few extra steps.

==== Setup ====
Before you use this for the first time, do the following steps:

*SSH into Wright and run the following to install and unpack the VSCode CLI: 
<pre>curl -Lk 'https://code.visualstudio.com/sha/download?build=stable&os=cli-alpine-x64' --output vscode_cli.tar.gz
tar -xf vscode_cli.tar.gz
</pre>

*Add the <code>code</code> executable to your PATH by adding the following to your <code>.bashrc</code> file: <code>export PATH="~:$PATH"</code><pre>echo 'export PATH="$PATH:/path/to/code"' >> ~/.bashrc
</pre>
*In VSCode, install the <code>Remote - Tunnels</code> extension, then use the VSCode SSH feature to connect to Wright and make sure to install all the local VS Code extensions on the remote machine as well (e.g., Python, Jupyter, ...)

==== Running a session ====
*In the Wright headnode, launch a Slurm interactive session with <code>srun</code>, following the instructions above.

*Inside the interactive session, run <code>code tunnel</code> and follow the instructions. The first time the system will ask you to name the tunnel and authorize the process via an authenticator, I suggest using GitHub since it usually works quite smoothly. Subsequent tunnels might still ask you to authorize their creation by typing an alphanumerical sequence into a link that appears in the terminal. Once you’ve done that, the tunnel is successfully created.
*From VSCode, select <code>Connect to Tunnel</code>, then choose the authentication method you previously used and you will see the tunnel appear in the dropdown. Select it and you will land directly into the interactive session we just created
*You will now be able to run code and, e.g., Jupyter Notebooks using the compute node allocated by the Slurm interactive session. VS Code will be disconnected when the Slurm session ends, such as when it reaches its time limit.
Note that if you need to use a conda environment, you'll have to configure VS Code to use the Python executable you found above, since conda is not installed on the compute nodes.

=== SSH to Compute Nodes ===
The ability to SSH directly to compute nodes is disabled. You can run the command below to start a bash terminal on a compute node when running a batch job. This allows for the same functionality as SSH.<syntaxhighlight lang="bash">
srun --jobid=yourjobidhere --pty /bin/bash
</syntaxhighlight>

== Monitoring cluster usage ==
The [https://util.stat.cmu.edu/grafana/d/Oxed_c6Wz/nvidia-wright-gpu-metrics NVIDIA Wright GPU Metrics Dashboard] shows graphs of GPU usage across the cluster. At the top left, you can select which node to graph, so viewing your department's node will let you see how GPUs are being used.

If you're developing a long-running job, we recommend checking this dashboard while your job is running. It is easy to accidentally write PyTorch or Tensorflow code that '''doesn't actually use the GPU''', so check the graphs to see if the expected number of GPUs are in used based on your job's requirements and the other jobs listed in <code>squeue</code>. If they are not, get a terminal on the compute node as described above and run <code>nvidia-smi</code> to get GPU usage information, including which scripts are accessing the GPU. If your script doesn't show up, you can debug it to figure out why no GPUs are being used.

== Using scratch storage ==
Each compute node (but not the head node) has several terabytes of scratch space. This is fast SSD storage that is directly attached to the node, rather than being accessed over the network (like all files in <code>/home</code> are). This is useful if your jobs need to write large temporary files: reading and writing the scratch space is faster than reading and writing files in <code>/home</code>.

Your home directory contains a symlink to the scratch space at <code>~/scratch</code>. To use it, create a directory inside named after your username:
mkdir ~/scratch/yourusername
Then you can read and write files in that directory.

'''Warnings:'''

* The scratch space is meant for '''temporary''' storage during a job. It may be cleared out during cluster maintenance or if the disk is full.
* The scratch drives are much smaller than the huge networked disk used for <code>/home</code>, so be careful what you store in it, and regularly delete files you don't need.
* Each node has separate scratch disks, so data you write on one node is '''not''' available from any other node, or from the head node.

The scratch space is thus not for final model outputs or results. It is ideal for things like saving checkpoints while your job is running, dumping data to debug a job, saving output from one script to be immediately used by another script, etc.

Department GPU Servers

2026-03-05T19:19:23Z

Areinhar:

The department has purchased and installed 3 new GPU servers that are physically located in Baker Hall. They have 64GB of RAM, 32 core processors and NVIDIA RTX 4090 GPU cards. You can log into them just as you would with the [[Linux_and_department_servers|hydras]], with your stats ID and password. They are named:

ghidorah1.stat.cmu.edu

ghidorah2.stat.cmu.edu

ghidorah3.stat.cmu.edu

* ghidorah3 is reserved for MSCF student capstone projects in the fall.

All of them share the same file system with the hydras and leviathans, which means that whatever you install, save and configure on one of the machines above will also be available in one of the hydras, and viceversa. 

A simple terminal command to check GPU usage is  <code>nvidia-smi</code>, which shows several pieces of information about the GPU and the corresponding CUDA drivers installed. This command also supports checks at regular time intervals (in seconds). For example, to check usage every 2 seconds use <code> nvidia-smi -l 2</code>

 

Department Servers

2026-03-03T13:17:38Z

Areinhar: /* Using the Servers */ fix typo

== Hydras ==
The Hydras are virtual based serves hosted in the Campus Cloud. They offer slightly higher stability and are better for smaller computing jobs.

Hardware Specifications

* Intel Xeon CPU
** 12 cores @2.6Ghz
* 64GB of RAM (128GB on hydra7 and hydra8)
* NAS used for /home storage
Servers = Hydra1-8

ssh yourusername@hydra1.stat.cmu.edu

[https://info.stat.cmu.edu/grafana/d/beuw1q6y3dgjkc/hydra-cluster?orgId=1 Grafana Monitoring]
== Ghidorahs ==
The Ghidorahs are physical servers located throughout Baker Hall. They are designed for lighter GPU jobs.

Hardware Specifications

* Core i9-13900KF CPU
** 8 performance cores @5.4Ghz
** 16 Efficiency cores @4.3Ghz
* 64GB of RAM
* RTX 4090 24GB of video memory
* NAS used for /home storage

[https://info.stat.cmu.edu/grafana/d/edkar1rgy3l6od/ghidorah-cluster?orgId=1 Grafana Monitoring]
== Leviathans ==
The Leviathans are physical servers located throughout Baker Hall and the Cyert Data Center. They are designed for heavy CPU and RAM jobs.

Hardware Specifications

* AMD Ryzen 9 9950x
** 16 cores @5.7Ghz
* 128GB of RAM (Leviathan1-6)
* 196GB of RAM (Leviathan7-11)
* 4TB of local NVME storage
** Mounted at /scratch
** Data stored in this location will only be accessible on the original server.
* NAS used for /home storage
Servers = Leviathan1-11

ssh yourusername@leviathan1.stat.cmu.edu

* You can use "levi" in place of "leviathan"
** ''ssh yourusername@levi1.stat.cmu.edu''

[https://info.stat.cmu.edu/grafana/d/asjkldfbnvajdsfh213/leviathan-cluster?orgId=1 Grafana Monitor]

== Storage ==
Each server's "/home" directory located on the department's NAS. This means your files will be accessible on every server.

The NAS is a total of 100TB. This space is shared and monitored for fair usage. If you store large amounts of data, you may be asked if it is all necessary.
To see available space on the NAS, run the command below in a terminal on the server.<syntaxhighlight lang="bash">
df -h | grep "/home"
</syntaxhighlight>To see your total usage run the command below. ''It may take a few minutes to finish''<syntaxhighlight lang="bash">
du -sh ~
</syntaxhighlight>

== Using the Servers ==

Accessing the servers requires using an SSH compatible terminal. Applications such as "Command Prompt" on Windows or "Terminal" on MacOS can achieve this.
To SSH to a server, open either app listed previously and enter the SSH command shown below. Be sure to use your StatDS computing account.<syntaxhighlight lang="bash">
ssh statdsusername@leviathan11.stat.cmu.edu
</syntaxhighlight>In the line above, replace "hydra1" with the server you want to connect to.
{| class="wikitable"
|+
Servers
!Hydras
!Ghidorahs
!Leviathans
|-
|hydra1
|ghidorah1
|leviathan1
|-
|hydra2
|ghidorah2
|leviathan2
|-
|hydra3
|ghidorah3
|leviathan3
|-
|hydra4
|
|leviathan4
|-
|hydra5
|
|leviathan5
|-
|hydra6
|
|leviathan6
|-
|hydra7
|
|leviathan7
|-
|
|
|leviathan8
|-
|
|
|leviathan9
|-
|
|
|leviathan10
|-
|
|
|leviathan11
|}

== Software ==
The servers within Statistics & Data Science offer a variety of software. Sudo access, also referred to as the ability to install software, on department servers is not allowed. If a software package you would like to use is not listed below, please contact remarks@stat.cmu.edu.
{| class="wikitable"
|+
!
!Hydras
!Ghidorahs
!Leviathans
|-
|Python
|X
|X
|X
|-
|R
|X
|X
|X
|-
|Apptainer
|X
|X
|X
|-
|Quarto
|X
|
|X
|-
|Gurobi
|X
|
|
|-
|Rstudio-Server
|X
|
|X
|-
|Matlab
|X
|
|X
|-
|Julia
|X
|
|X
|}

Wright GPU Cluster

2026-01-29T20:19:59Z

Areinhar: /* SSH to Compute Nodes */ fix typo in arg

The department has access to a powerful GPU-based cluster named [https://en.wikipedia.org/wiki/Sewall_Wright Wright] which is hosted here at CMU. This is a 4-node and growing cluster that is a shared resource for Statistics & Data Science, Philosophy, and other departments in Dietrich College. Each department has priority access to its own compute node, but will be able to share the resources from other departments when not in use.

== Access requests ==
This resource is free to use to users in the Departments of Statistics & Data Science, Philosophy, and CMIST. These departments set policies determining which of their users are eligible. Users outside these departments must be sponsored by someone in the eligible departments, as part of a project or collaboration.

To request access, complete the [https://docs.google.com/forms/d/e/1FAIpQLSftCsplE06whGocn62DhwC6NpDSgZ-DMmUacrQID_slIfg6sA/viewform access request form] and our team will be in touch to discuss your request and onboard you.

== Cluster setup ==
The cluster currently consists of 4 servers:
*The head node: You log in to this to write code and run jobs. It has 128 GB of RAM, 16 cores, and '''no GPU'''.
*The compute nodes: Jobs run on these. Each has 256 CPU cores, 8 NVIDIA L40 GPU cards with 48 GB of video RAM each, approximately 1 TB of RAM, and a total of approximately 3 TB of local data/scratch space.
*The file server: Holds 50TB of storage. The head node and compute nodes connect via fiberoptic networking. Your home directory is hosted on the file server, so the same files are accessible on the head node and all compute nodes.
The cluster is currently partitioned. One compute node is reserved for each department involved.

The file server has quotas enabled so that each department gets 1/3rd of the available storage. If a department user tries to add files beyond the quota, the writes will be rejected, and nobody in that department will be able to add files until space is freed. To check the quotas and your own usage, [https://util.stat.cmu.edu/quotareport/ view the quota report].

== Connecting and setting up your environment ==

=== SSH ===

From your terminal, type <code>ssh <YourStatID>@wright.hss.cmu.edu</code> . You can also avoid remembering this by adding the following snippet to your local <code>~/.ssh/config</code> file:
<pre>Host wright
Hostname wright.hss.cmu.edu
User YourStatID
</pre>

You can then connect by simply typing <code>ssh wright</code>.

This connects you to the '''head node.''' Here you can upload your data files, set up your scripts, and test things.

'''Do not run resource-intensive tasks directly on the head node'''. It doesn't have any GPUs, and it has fewer resources than the compute nodes. You can do quick tests to make sure your scripts are set up correctly, but submit all other jobs through Slurm.

=== Uploading files ===
To upload scripts or data files, you can use <code>scp</code> from your local computer (Mac or Linux); on Windows, you can get the Tectia SSH graphical client [https://www.cmu.edu/computing/services/endpoint/software/access.html free through CMU].

You can also connect through the [https://code.visualstudio.com/docs/remote/ssh Remote SSH feature of VS Code] if you like. This is useful to 

*navigate the file system from a proper GUI and 
*modify code/files/folder in place if you are not familiar with the command line
Files uploaded to your home directory will be accessible on the head node and all compute nodes, so they can be freely used in compute jobs.

=== Using Conda ===
The head node has Anaconda installed for managing Python packages and system libraries; see [[Python virtual environments]] for information on how to use it (but note it's '''already installed''', so you don't need to install it from scratch!).

However, the compute nodes '''do not''' have conda, so your slurm jobs can't use <code>conda activate</code> or other familiar commands. Instead, set up your environment on the head node, activate it, install the packages you need, and so on. Then run <code>which python</code> to get the path to the Python executable in your environment. For instance:<syntaxhighlight lang="bash">
$ which python
/home/export/areinhar/.conda/envs/llm/bin/python
</syntaxhighlight>This is the path you'll use in your scripts (see below).

=== Open OnDemand ===

Open OnDemand is a graphical interface where you can run jobs, interactive applications, and manage files. More information can be found [[Open OnDemand Wright|here]].

== Job scheduling and prioritization ==

The GPU cluster manages jobs via [[Main_Page/slurm|Slurm]]. Slurm is a job management system that deals with job prioritization and the allocation of resources. On high-performance computers (e.g., PSC and our GPU cluster) Slurm helps allocate resources so multiple jobs can be run quickly. Users run programs by submitting scripts to the Slurm job scheduler, and the scheduler runs them when resources are available.
=== Queue status ===

To check what jobs are running, use <code>squeue</code>. The following options list more helpful information than the default (though you need a wide terminal to read it all):<syntaxhighlight lang="bash">
$ squeue -O "JobID:7,UserName:10,Partition:12,TimeUsed:12,ReasonList,MinCPUS:9,MinMemory:12,tres-per-node,StartTime,EndTime"
JOBID USER PARTITION TIME NODELIST(REASON) MIN_CPUS MIN_MEMORY TRES_PER_NODE START_TIME END_TIME
36229 ebensen all 0:00 (DependencyNeverSati16 128G N/A N/A N/A
36237 ajshen all 12:35:57 n02 2 64G gres:gpu:1 2026-01-26T21:55:23 2026-01-28T21:55:23
36152 msezgin all 1-02:32:04 n03 1 64G gres:gpu:1 2026-01-26T07:59:16 2026-01-28T07:59:16
36132 tsukiant cmist_condo 1-11:00:02 n03 16 8G gres:gpu:1 2026-01-25T23:31:18 2026-01-27T23:31:18
36100 tsukiant cmist_condo 1-17:32:39 n03 16 8G gres:gpu:1 2026-01-25T16:58:41 2026-01-27T16:58:41
36090 tsukiant cmist_condo 1-18:22:51 n03 16 8G gres:gpu:1 2026-01-25T16:08:29 2026-01-27T16:08:29
36247 jwalches statds 42:38 n01 40 15G gres:gpu:3 2026-01-27T09:48:42 2026-01-29T09:48:42
36051 xiangchs statds_condo7:05:24 n01 64 128G gres:gpu:4 2026-01-27T03:25:56 2026-01-29T03:25:56
</syntaxhighlight>

Here we see there are two jobs running on node 1, one on node 2, four on node 3, and one job that is not running because it was submitted with requirements that will not be met. The listed end times are based on the time limit specified when the job was submitted, so the jobs may end sooner. If you submit a job that is waiting for resources, the reason will be listed as <code>(Resources)</code> or <code>(Priority)</code>, and after a few minutes Slurm will list a start time for your job based on when other jobs will finish and free up resources.

To check what resources are currently in use, use <code>sinfo</code>:<syntaxhighlight lang="bash">
$ sinfo -O "NodeList:10,CPUsState:16,Memory:9,FreeMem:10,Gres:20,GresUsed:30"
NODELIST CPUS(A/I/O/T) MEMORY FREE_MEM GRES GRES_USED
n01 64/192/0/256 1000000 221943 gpu:nvidia-l40:8 gpu:nvidia-l40:8(IDX:0-7)
n02 32/224/0/256 1000000 225575 gpu:nvidia-l40:8 gpu:nvidia-l40:0(IDX:N/A)
</syntaxhighlight>This shows that node 1 (Statistics & Data Science) has 64 CPUs allocated and 192 idle, out of 256 total. There is 1TB of RAM, of which 222 GB is free. There are 8 NVIDIA L40 GPus, all 8 of which are in use. Hence if we submit a job to the statds partition requesting any GPUs, it will have to wait until some become free. Meanwhile, node 2 (Philosophy) has no GPUs in use.

=== Partitions and condos ===
The cluster is divided into three partitions:

* <code>statds</code> (Statistics & Data Science)
* <code>phil</code> (Philosophy)
* <code>cmist</code> (CMIST)

Jobs are submitted to a partition. '''Users must submit jobs to their department's partition''' -- jobs submitted to the wrong partition will be rejected.

There are several other special partitions:

* <code>all</code> requests any available node from any partition. Thus a Philosophy user can run a job on the CMIST node, or vice versa.
* <code>statds_condo</code>, <code>phil_condo</code>, and <code>cmist_condo</code> are "condo" partitions. Any user can submit to any of these partitions.

These special partitions are like holiday condos: any user can access them, but '''with a lower priority''' than the owner (the host department). Jobs running on them are also preemptible. For example, if a Philosophy user submits to <code>statds_condo</code>, their job will run only when Statistics users are not using the resources. If a Statistics user submits a large job after yours, your job might be terminated, and will automatically run again when resources become available. The <code>all</code> partition is similar, but for whichever node happens to be free.

=== Job prioritization ===
Slurm runs jobs based on the resources they request and what is available. If resources aren't available for your job, slurm will put it in the queue and wait until other jobs finish. However, it's not simply a first-in first-out queue -- Slurm attempts to arrange the jobs so every job can run, even if that means running them out of order. Slurm also adjusts each user's priority based on how many jobs they submit, so that one user can't hog the cluster by submitting dozens of jobs; if one user has many jobs waiting, theirs will be reduced in priority so other users can run jobs as well.

Examples:

* Someone submits Job A that requests 6 GPUs. Later, you submit Job B that requests 4 GPUs on the same partition. The compute node only has 8 GPUs, so Job B will be queued until Job A completes.
* Someone submits Job A that requests 6 GPUs. Later, you submit Job B that requests 2 GPUs. Job B can run immediately.
* You submit a job that requires all 8 GPUs on a node, but 4 are currently in use. Your job is put in the queue and scheduled to run when Slurm knows the current jobs will finish (because of their maximum time limit). Meanwhile, someone submits a job using 2 GPUs and a short time limit. Slurm lets it run immediately, because it knows it will be done before your job is scheduled to run. Other jobs may be put off until after yours runs, so yours can get the 8 GPUs it needs.
* You submit 50 jobs that each require 1 GPU. The node has only 8 GPUs, but several are in use. Several of your jobs run, but the others are queued for later. Other users can submit jobs while yours are in the queue, and some of their jobs may run before some of yours, for fairness.
* You submit a job to a condo partition. The partition is currently full, so your job is queued. Other users from that department submit new jobs. As they are higher priority than you, their jobs run first.
You can view the currently running or queued jobs by running <code>squeue</code> on the head node, as described above. The <code>NODELIST(REASON)</code> column will indicate which jobs are queued by giving special codes:

* <code>(Resources)</code> means there are not enough resources (CPUs, memory, or GPUs) available for the job. It will run once the resources are free.
* <code>(Priority)</code> means resources are available, but Slurm wants to run a higher-priority job first. For example, maybe a large job has been waiting for resources for two days before you submit your job; Slurm may make your job wait so the large job can start first.

=== Job etiquette ===
The cluster is shared between users and departments. Therefore:

* Request the minimum resources required for your job, so other users can access other resources.
* If you need to submit a large number of jobs and aren't in a hurry, consider using your department's condo partition or <code>all</code>. Using <code>all</code> lets you get all available resources, but at lower priority, so other users can run jobs as well and you don't hog the cluster.
* Watch your disk space usage. With large deep models it is easy to use many terabytes of storage, and each department has a quota. [https://util.stat.cmu.edu/quotareport/ You can check the quota and your own usage here.]

== Setting up jobs ==

=== Submitting batch jobs ===
The best way to run jobs is by writing a shell script specifying the Slurm options. To launch a batch job from a script you will need two files:

*the actual script in your favorite programming language that does whatever computation you want
*a bash script that requests the resources you need from Slurm and gives the commands to launch your job.
A typical bash script might look like this:<pre>#!/bin/bash
# specify partition to run in (statds, phil, or cmist)
#SBATCH --partition statds
#SBATCH --ntasks 1
#SBATCH --job-name=nameofyourjob

# number of CPUs you need:
#SBATCH --cpus-per-task 1

# amount of memory per CPU
# (total allocated is CPUs * memory)
#SBATCH --mem-per-cpu 4G

# number of GPUs you need
#SBATCH --gres=gpu:1

# maximum time limit for job, HH:MM:SS (up to 48 hours max, 48:00:00)
#SBATCH --time=01:00:00

# email you when job is finished or errors out:
#SBATCH --mail-user=YourAndrewID@andrew.cmu.edu
#SBATCH --mail-type=END,FAIL

## Choose one:

# to run a Python script in a Conda environment called ENV
srun /home/export/YOURANDREWID/.conda/envs/ENV/bin/python YourScript.py

# to run an R script:
srun Rscript YourScript.R
</pre>Adjust the number of CPUs, number of GPUs, memory usage, and time limit based on the needs of your script. Requesting more than you need may result in your job having to wait for resources to free up; requesting a long time you don't need will block slurm from scheduling other jobs for when yours will be done. (And if you request more resources than exist on the compute nodes, your job will be queued forever while it waits for resources to appear.)

'''Note:''' Slurm batch jobs are limited to a maximum duration of 48 hours. Jobs running longer will be killed.

To run the job, SSH into the head node and navigate to the directory with your scripts. Then run:<syntaxhighlight lang="bash">
sbatch example.sh
</syntaxhighlight>where <code>example.sh</code> is the name of your bash script.

You will see a message like "Submitted batch job 1209" when your job is submitted. Any printed output from your script will automatically be redirected to a file based on the job number -- in this case, <code>slurm-1209.out</code>. You can use <code>tail -f slurm-1209.out</code> to view the output live and monitor your script's progress.

=== Selecting the right resources ===
In the Slurm script above, you can choose the number of tasks, the number of GPUs, the memory per CPU, the number of GPUs, and the time limit. Choose these values wisely:

* You generally want '''1 task''' (<code>--ntasks 1</code>). Asking for multiple tasks will cause srun to run multiple copies of your script simultaneously. If you need to run the same script with 10 different parameter values, this can be useful: request 10 tasks, and read the <code>SLURM_PROCID</code> environment variable in your script. It will be numbered 0-9 (for 10 tasks), so you can use this to set the parameter values for each parallel run.
* Set the number of CPUs based on your script. If your script is entirely single-threaded, ask for 1. If you use parallel processing (such as with multiprocessing in Python or the parallel package in R), use a multithreaded package, or spawn many parallel programs in your job, request more.
* Request the minimum amount of memory (RAM) needed for your script. This is per-CPU, so requesting a lot of CPUs and RAM can easily mean you request more RAM than exists on the node.
* Start with 1 GPU, adding more if your job requires more GPU memory (VRAM) than is on one GPU, or if you explicitly write your script to use more. If you do not use GPUs, request 0 so you aren't allocated a GPU that nobody else can use while your job runs.
* Set the time limit a little longer than you expect your script to run. Slurm uses the time limit when scheduling, so always choosing the maximum (48 hours) will backfire: Slurm will assume your job will take a long time and will run other things first. If you know your job can only take an hour, setting, say, a 01:30:00 time limit will let Slurm fit your job in before other long jobs in the queue. But of course if you choose too low and your job exceeds the limit, it will be killed, so leave yourself some extra time.

=== Interactive mode ===

You can also run interactive sessions that allow you to, say, run an R or Python console and type commands and immediately view the output. From the head node, run a command like:<syntaxhighlight lang="bash">
# remember to edit to correct partition:
srun --partition statds --ntasks 1 \
--cpus-per-task 16 --mem-per-cpu 4G \
--gres=gpu:1 --time=01:00:00 --pty \
bash
</syntaxhighlight>This runs a job that opens a bash shell on the compute node, and connects you directly to this job. You can now run bash commands, open Python, or run R scripts. Note your interactive session has a time limit and will be killed when it elapses, so set the time limit carefully and choose the resource amounts in the command according to your needs.

Interactive sessions are limited to '''two hours maximum'''. You cannot request a longer time limit. This is because interactive sessions are inefficient, with the cluster spending most of its time waiting for you to type something instead of running code. For large tasks, submit a batch job.

=== Interactive via VS Code ===
VS Code supports SSH and tunneling, which allows it to connect to a server and run commands. This allows you to use the Jupyter Notebook support built into VS Code to interact with the GPU cluster. As this uses an interactive session, it's subject to the same '''two-hour time limit'''. Hence notebooks should be used only for exploration and debugging, not large jobs.

To use this, we need to connect to the head node, run an interactive Slurm session, and then connect VS Code to that Slurm session so it can use it. This takes a few extra steps.

==== Setup ====
Before you use this for the first time, do the following steps:

*SSH into Wright and run the following to install and unpack the VSCode CLI: 
<pre>curl -Lk 'https://code.visualstudio.com/sha/download?build=stable&os=cli-alpine-x64' --output vscode_cli.tar.gz
tar -xf vscode_cli.tar.gz
</pre>

*Add the <code>code</code> executable to your PATH by adding the following to your <code>.bashrc</code> file: <code>export PATH="~:$PATH"</code><pre>echo 'export PATH="$PATH:/path/to/code"' >> ~/.bashrc
</pre>
*In VSCode, install the <code>Remote - Tunnels</code> extension, then use the VSCode SSH feature to connect to Wright and make sure to install all the local VS Code extensions on the remote machine as well (e.g., Python, Jupyter, ...)

==== Running a session ====
*In the Wright headnode, launch a Slurm interactive session with <code>srun</code>, following the instructions above.

*Inside the interactive session, run <code>code tunnel</code> and follow the instructions. The first time the system will ask you to name the tunnel and authorize the process via an authenticator, I suggest using GitHub since it usually works quite smoothly. Subsequent tunnels might still ask you to authorize their creation by typing an alphanumerical sequence into a link that appears in the terminal. Once you’ve done that, the tunnel is successfully created.
*From VSCode, select <code>Connect to Tunnel</code>, then choose the authentication method you previously used and you will see the tunnel appear in the dropdown. Select it and you will land directly into the interactive session we just created
*You will now be able to run code and, e.g., Jupyter Notebooks using the compute node allocated by the Slurm interactive session. VS Code will be disconnected when the Slurm session ends, such as when it reaches its time limit.
Note that if you need to use a conda environment, you'll have to configure VS Code to use the Python executable you found above, since conda is not installed on the compute nodes.

=== SSH to Compute Nodes ===
The ability to SSH directly to compute nodes is disabled. You can run the command below to start a bash terminal on a compute node when running a batch job. This allows for the same functionality as SSH.<syntaxhighlight lang="bash">
srun --jobid=yourjobidhere --pty /bin/bash
</syntaxhighlight>

== Monitoring cluster usage ==
The [https://util.stat.cmu.edu/grafana/d/Oxed_c6Wz/nvidia-wright-gpu-metrics NVIDIA Wright GPU Metrics Dashboard] shows graphs of GPU usage across the cluster. At the top left, you can select which node to graph, so viewing your department's node will let you see how GPUs are being used.

If you're developing a long-running job, we recommend checking this dashboard while your job is running. It is easy to accidentally write PyTorch or Tensorflow code that '''doesn't actually use the GPU''', so check the graphs to see if the expected number of GPUs are in used based on your job's requirements and the other jobs listed in <code>squeue</code>. If they are not, get a terminal on the compute node as described above and run <code>nvidia-smi</code> to get GPU usage information, including which scripts are accessing the GPU. If your script doesn't show up, you can debug it to figure out why no GPUs are being used.

== Using scratch storage ==
Each compute node (but not the head node) has several terabytes of scratch space. This is fast SSD storage that is directly attached to the node, rather than being accessed over the network (like all files in <code>/home</code> are). This is useful if your jobs need to write large temporary files: reading and writing the scratch space is faster than reading and writing files in <code>/home</code>.

Your home directory contains a symlink to the scratch space at <code>~/scratch</code>. To use it, create a directory inside named after your username:
mkdir ~/scratch/yourusername
Then you can read and write files in that directory.

'''Warnings:'''

* The scratch space is meant for '''temporary''' storage during a job. It may be cleared out during cluster maintenance or if the disk is full.
* The scratch drives are much smaller than the huge networked disk used for <code>/home</code>, so be careful what you store in it, and regularly delete files you don't need.
* Each node has separate scratch disks, so data you write on one node is '''not''' available from any other node, or from the head node.

The scratch space is thus not for final model outputs or results. It is ideal for things like saving checkpoints while your job is running, dumping data to debug a job, saving output from one script to be immediately used by another script, etc.

Wright GPU Cluster

2026-01-27T18:32:20Z

Areinhar: /* Partitions and condos */

The department has access to a powerful GPU-based cluster named [https://en.wikipedia.org/wiki/Sewall_Wright Wright] which is hosted here at CMU. This is a 4-node and growing cluster that is a shared resource for Statistics & Data Science, Philosophy, and other departments in Dietrich College. Each department has priority access to its own compute node, but will be able to share the resources from other departments when not in use.

== Access requests ==
This resource is free to use to users in the Departments of Statistics & Data Science, Philosophy, and CMIST. These departments set policies determining which of their users are eligible. Users outside these departments must be sponsored by someone in the eligible departments, as part of a project or collaboration.

To request access, complete the [https://docs.google.com/forms/d/e/1FAIpQLSftCsplE06whGocn62DhwC6NpDSgZ-DMmUacrQID_slIfg6sA/viewform access request form] and our team will be in touch to discuss your request and onboard you.

== Cluster setup ==
The cluster currently consists of 4 servers:
*The head node: You log in to this to write code and run jobs. It has 128 GB of RAM, 16 cores, and '''no GPU'''.
*The compute nodes: Jobs run on these. Each has 256 CPU cores, 8 NVIDIA L40 GPU cards with 48 GB of video RAM each, approximately 1 TB of RAM, and a total of approximately 3 TB of local data/scratch space.
*The file server: Holds 50TB of storage. The head node and compute nodes connect via fiberoptic networking. Your home directory is hosted on the file server, so the same files are accessible on the head node and all compute nodes.
The cluster is currently partitioned. One compute node is reserved for each department involved.

The file server has quotas enabled so that each department gets 1/3rd of the available storage. If a department user tries to add files beyond the quota, the writes will be rejected, and nobody in that department will be able to add files until space is freed. To check the quotas and your own usage, [https://util.stat.cmu.edu/quotareport/ view the quota report].

== Connecting and setting up your environment ==

=== SSH ===

From your terminal, type <code>ssh <YourStatID>@wright.hss.cmu.edu</code> . You can also avoid remembering this by adding the following snippet to your local <code>~/.ssh/config</code> file:
<pre>Host wright
Hostname wright.hss.cmu.edu
User YourStatID
</pre>

You can then connect by simply typing <code>ssh wright</code>.

This connects you to the '''head node.''' Here you can upload your data files, set up your scripts, and test things.

'''Do not run resource-intensive tasks directly on the head node'''. It doesn't have any GPUs, and it has fewer resources than the compute nodes. You can do quick tests to make sure your scripts are set up correctly, but submit all other jobs through Slurm.

=== Uploading files ===
To upload scripts or data files, you can use <code>scp</code> from your local computer (Mac or Linux); on Windows, you can get the Tectia SSH graphical client [https://www.cmu.edu/computing/services/endpoint/software/access.html free through CMU].

You can also connect through the [https://code.visualstudio.com/docs/remote/ssh Remote SSH feature of VS Code] if you like. This is useful to 

*navigate the file system from a proper GUI and 
*modify code/files/folder in place if you are not familiar with the command line
Files uploaded to your home directory will be accessible on the head node and all compute nodes, so they can be freely used in compute jobs.

=== Using Conda ===
The head node has Anaconda installed for managing Python packages and system libraries; see [[Python virtual environments]] for information on how to use it (but note it's '''already installed''', so you don't need to install it from scratch!).

However, the compute nodes '''do not''' have conda, so your slurm jobs can't use <code>conda activate</code> or other familiar commands. Instead, set up your environment on the head node, activate it, install the packages you need, and so on. Then run <code>which python</code> to get the path to the Python executable in your environment. For instance:<syntaxhighlight lang="bash">
$ which python
/home/export/areinhar/.conda/envs/llm/bin/python
</syntaxhighlight>This is the path you'll use in your scripts (see below).

=== Open OnDemand ===

Open OnDemand is a graphical interface where you can run jobs, interactive applications, and manage files. More information can be found [[Open OnDemand Wright|here]].

== Job scheduling and prioritization ==

The GPU cluster manages jobs via [[Main_Page/slurm|Slurm]]. Slurm is a job management system that deals with job prioritization and the allocation of resources. On high-performance computers (e.g., PSC and our GPU cluster) Slurm helps allocate resources so multiple jobs can be run quickly. Users run programs by submitting scripts to the Slurm job scheduler, and the scheduler runs them when resources are available.
=== Queue status ===

To check what jobs are running, use <code>squeue</code>. The following options list more helpful information than the default (though you need a wide terminal to read it all):<syntaxhighlight lang="bash">
$ squeue -O "JobID:7,UserName:10,Partition:12,TimeUsed:12,ReasonList,MinCPUS:9,MinMemory:12,tres-per-node,StartTime,EndTime"
JOBID USER PARTITION TIME NODELIST(REASON) MIN_CPUS MIN_MEMORY TRES_PER_NODE START_TIME END_TIME
36229 ebensen all 0:00 (DependencyNeverSati16 128G N/A N/A N/A
36237 ajshen all 12:35:57 n02 2 64G gres:gpu:1 2026-01-26T21:55:23 2026-01-28T21:55:23
36152 msezgin all 1-02:32:04 n03 1 64G gres:gpu:1 2026-01-26T07:59:16 2026-01-28T07:59:16
36132 tsukiant cmist_condo 1-11:00:02 n03 16 8G gres:gpu:1 2026-01-25T23:31:18 2026-01-27T23:31:18
36100 tsukiant cmist_condo 1-17:32:39 n03 16 8G gres:gpu:1 2026-01-25T16:58:41 2026-01-27T16:58:41
36090 tsukiant cmist_condo 1-18:22:51 n03 16 8G gres:gpu:1 2026-01-25T16:08:29 2026-01-27T16:08:29
36247 jwalches statds 42:38 n01 40 15G gres:gpu:3 2026-01-27T09:48:42 2026-01-29T09:48:42
36051 xiangchs statds_condo7:05:24 n01 64 128G gres:gpu:4 2026-01-27T03:25:56 2026-01-29T03:25:56
</syntaxhighlight>

Here we see there are two jobs running on node 1, one on node 2, four on node 3, and one job that is not running because it was submitted with requirements that will not be met. The listed end times are based on the time limit specified when the job was submitted, so the jobs may end sooner. If you submit a job that is waiting for resources, the reason will be listed as <code>(Resources)</code> or <code>(Priority)</code>, and after a few minutes Slurm will list a start time for your job based on when other jobs will finish and free up resources.

To check what resources are currently in use, use <code>sinfo</code>:<syntaxhighlight lang="bash">
$ sinfo -O "NodeList:10,CPUsState:16,Memory:9,FreeMem:10,Gres:20,GresUsed:30"
NODELIST CPUS(A/I/O/T) MEMORY FREE_MEM GRES GRES_USED
n01 64/192/0/256 1000000 221943 gpu:nvidia-l40:8 gpu:nvidia-l40:8(IDX:0-7)
n02 32/224/0/256 1000000 225575 gpu:nvidia-l40:8 gpu:nvidia-l40:0(IDX:N/A)
</syntaxhighlight>This shows that node 1 (Statistics & Data Science) has 64 CPUs allocated and 192 idle, out of 256 total. There is 1TB of RAM, of which 222 GB is free. There are 8 NVIDIA L40 GPus, all 8 of which are in use. Hence if we submit a job to the statds partition requesting any GPUs, it will have to wait until some become free. Meanwhile, node 2 (Philosophy) has no GPUs in use.

=== Partitions and condos ===
The cluster is divided into three partitions:

* <code>statds</code> (Statistics & Data Science)
* <code>phil</code> (Philosophy)
* <code>cmist</code> (CMIST)

Jobs are submitted to a partition. '''Users must submit jobs to their department's partition''' -- jobs submitted to the wrong partition will be rejected.

There are several other special partitions:

* <code>all</code> requests any available node from any partition. Thus a Philosophy user can run a job on the CMIST node, or vice versa.
* <code>statds_condo</code>, <code>phil_condo</code>, and <code>cmist_condo</code> are "condo" partitions. Any user can submit to any of these partitions.

These special partitions are like holiday condos: any user can access them, but '''with a lower priority''' than the owner (the host department). Jobs running on them are also preemptible. For example, if a Philosophy user submits to <code>statds_condo</code>, their job will run only when Statistics users are not using the resources. If a Statistics user submits a large job after yours, your job might be terminated, and will automatically run again when resources become available. The <code>all</code> partition is similar, but for whichever node happens to be free.

=== Job prioritization ===
Slurm runs jobs based on the resources they request and what is available. If resources aren't available for your job, slurm will put it in the queue and wait until other jobs finish. However, it's not simply a first-in first-out queue -- Slurm attempts to arrange the jobs so every job can run, even if that means running them out of order. Slurm also adjusts each user's priority based on how many jobs they submit, so that one user can't hog the cluster by submitting dozens of jobs; if one user has many jobs waiting, theirs will be reduced in priority so other users can run jobs as well.

Examples:

* Someone submits Job A that requests 6 GPUs. Later, you submit Job B that requests 4 GPUs on the same partition. The compute node only has 8 GPUs, so Job B will be queued until Job A completes.
* Someone submits Job A that requests 6 GPUs. Later, you submit Job B that requests 2 GPUs. Job B can run immediately.
* You submit a job that requires all 8 GPUs on a node, but 4 are currently in use. Your job is put in the queue and scheduled to run when Slurm knows the current jobs will finish (because of their maximum time limit). Meanwhile, someone submits a job using 2 GPUs and a short time limit. Slurm lets it run immediately, because it knows it will be done before your job is scheduled to run. Other jobs may be put off until after yours runs, so yours can get the 8 GPUs it needs.
* You submit 50 jobs that each require 1 GPU. The node has only 8 GPUs, but several are in use. Several of your jobs run, but the others are queued for later. Other users can submit jobs while yours are in the queue, and some of their jobs may run before some of yours, for fairness.
* You submit a job to a condo partition. The partition is currently full, so your job is queued. Other users from that department submit new jobs. As they are higher priority than you, their jobs run first.
You can view the currently running or queued jobs by running <code>squeue</code> on the head node, as described above. The <code>NODELIST(REASON)</code> column will indicate which jobs are queued by giving special codes:

* <code>(Resources)</code> means there are not enough resources (CPUs, memory, or GPUs) available for the job. It will run once the resources are free.
* <code>(Priority)</code> means resources are available, but Slurm wants to run a higher-priority job first. For example, maybe a large job has been waiting for resources for two days before you submit your job; Slurm may make your job wait so the large job can start first.

=== Job etiquette ===
The cluster is shared between users and departments. Therefore:

* Request the minimum resources required for your job, so other users can access other resources.
* If you need to submit a large number of jobs and aren't in a hurry, consider using your department's condo partition or <code>all</code>. Using <code>all</code> lets you get all available resources, but at lower priority, so other users can run jobs as well and you don't hog the cluster.
* Watch your disk space usage. With large deep models it is easy to use many terabytes of storage, and each department has a quota. [https://util.stat.cmu.edu/quotareport/ You can check the quota and your own usage here.]

== Setting up jobs ==

=== Submitting batch jobs ===
The best way to run jobs is by writing a shell script specifying the Slurm options. To launch a batch job from a script you will need two files:

*the actual script in your favorite programming language that does whatever computation you want
*a bash script that requests the resources you need from Slurm and gives the commands to launch your job.
A typical bash script might look like this:<pre>#!/bin/bash
# specify partition to run in (statds, phil, or cmist)
#SBATCH --partition statds
#SBATCH --ntasks 1
#SBATCH --job-name=nameofyourjob

# number of CPUs you need:
#SBATCH --cpus-per-task 1

# amount of memory per CPU
# (total allocated is CPUs * memory)
#SBATCH --mem-per-cpu 4G

# number of GPUs you need
#SBATCH --gres=gpu:1

# maximum time limit for job, HH:MM:SS (up to 48 hours max, 48:00:00)
#SBATCH --time=01:00:00

# email you when job is finished or errors out:
#SBATCH --mail-user=YourAndrewID@andrew.cmu.edu
#SBATCH --mail-type=END,FAIL

## Choose one:

# to run a Python script in a Conda environment called ENV
srun /home/export/YOURANDREWID/.conda/envs/ENV/bin/python YourScript.py

# to run an R script:
srun Rscript YourScript.R
</pre>Adjust the number of CPUs, number of GPUs, memory usage, and time limit based on the needs of your script. Requesting more than you need may result in your job having to wait for resources to free up; requesting a long time you don't need will block slurm from scheduling other jobs for when yours will be done. (And if you request more resources than exist on the compute nodes, your job will be queued forever while it waits for resources to appear.)

'''Note:''' Slurm batch jobs are limited to a maximum duration of 48 hours. Jobs running longer will be killed.

To run the job, SSH into the head node and navigate to the directory with your scripts. Then run:<syntaxhighlight lang="bash">
sbatch example.sh
</syntaxhighlight>where <code>example.sh</code> is the name of your bash script.

You will see a message like "Submitted batch job 1209" when your job is submitted. Any printed output from your script will automatically be redirected to a file based on the job number -- in this case, <code>slurm-1209.out</code>. You can use <code>tail -f slurm-1209.out</code> to view the output live and monitor your script's progress.

=== Selecting the right resources ===
In the Slurm script above, you can choose the number of tasks, the number of GPUs, the memory per CPU, the number of GPUs, and the time limit. Choose these values wisely:

* You generally want '''1 task''' (<code>--ntasks 1</code>). Asking for multiple tasks will cause srun to run multiple copies of your script simultaneously. If you need to run the same script with 10 different parameter values, this can be useful: request 10 tasks, and read the <code>SLURM_PROCID</code> environment variable in your script. It will be numbered 0-9 (for 10 tasks), so you can use this to set the parameter values for each parallel run.
* Set the number of CPUs based on your script. If your script is entirely single-threaded, ask for 1. If you use parallel processing (such as with multiprocessing in Python or the parallel package in R), use a multithreaded package, or spawn many parallel programs in your job, request more.
* Request the minimum amount of memory (RAM) needed for your script. This is per-CPU, so requesting a lot of CPUs and RAM can easily mean you request more RAM than exists on the node.
* Start with 1 GPU, adding more if your job requires more GPU memory (VRAM) than is on one GPU, or if you explicitly write your script to use more. If you do not use GPUs, request 0 so you aren't allocated a GPU that nobody else can use while your job runs.
* Set the time limit a little longer than you expect your script to run. Slurm uses the time limit when scheduling, so always choosing the maximum (48 hours) will backfire: Slurm will assume your job will take a long time and will run other things first. If you know your job can only take an hour, setting, say, a 01:30:00 time limit will let Slurm fit your job in before other long jobs in the queue. But of course if you choose too low and your job exceeds the limit, it will be killed, so leave yourself some extra time.

=== Interactive mode ===

You can also run interactive sessions that allow you to, say, run an R or Python console and type commands and immediately view the output. From the head node, run a command like:<syntaxhighlight lang="bash">
# remember to edit to correct partition:
srun --partition statds --ntasks 1 \
--cpus-per-task 16 --mem-per-cpu 4G \
--gres=gpu:1 --time=01:00:00 --pty \
bash
</syntaxhighlight>This runs a job that opens a bash shell on the compute node, and connects you directly to this job. You can now run bash commands, open Python, or run R scripts. Note your interactive session has a time limit and will be killed when it elapses, so set the time limit carefully and choose the resource amounts in the command according to your needs.

Interactive sessions are limited to '''two hours maximum'''. You cannot request a longer time limit. This is because interactive sessions are inefficient, with the cluster spending most of its time waiting for you to type something instead of running code. For large tasks, submit a batch job.

=== Interactive via VS Code ===
VS Code supports SSH and tunneling, which allows it to connect to a server and run commands. This allows you to use the Jupyter Notebook support built into VS Code to interact with the GPU cluster. As this uses an interactive session, it's subject to the same '''two-hour time limit'''. Hence notebooks should be used only for exploration and debugging, not large jobs.

To use this, we need to connect to the head node, run an interactive Slurm session, and then connect VS Code to that Slurm session so it can use it. This takes a few extra steps.

==== Setup ====
Before you use this for the first time, do the following steps:

*SSH into Wright and run the following to install and unpack the VSCode CLI: 
<pre>curl -Lk 'https://code.visualstudio.com/sha/download?build=stable&os=cli-alpine-x64' --output vscode_cli.tar.gz
tar -xf vscode_cli.tar.gz
</pre>

*Add the <code>code</code> executable to your PATH by adding the following to your <code>.bashrc</code> file: <code>export PATH="~:$PATH"</code><pre>echo 'export PATH="$PATH:/path/to/code"' >> ~/.bashrc
</pre>
*In VSCode, install the <code>Remote - Tunnels</code> extension, then use the VSCode SSH feature to connect to Wright and make sure to install all the local VS Code extensions on the remote machine as well (e.g., Python, Jupyter, ...)

==== Running a session ====
*In the Wright headnode, launch a Slurm interactive session with <code>srun</code>, following the instructions above.

*Inside the interactive session, run <code>code tunnel</code> and follow the instructions. The first time the system will ask you to name the tunnel and authorize the process via an authenticator, I suggest using GitHub since it usually works quite smoothly. Subsequent tunnels might still ask you to authorize their creation by typing an alphanumerical sequence into a link that appears in the terminal. Once you’ve done that, the tunnel is successfully created.
*From VSCode, select <code>Connect to Tunnel</code>, then choose the authentication method you previously used and you will see the tunnel appear in the dropdown. Select it and you will land directly into the interactive session we just created
*You will now be able to run code and, e.g., Jupyter Notebooks using the compute node allocated by the Slurm interactive session. VS Code will be disconnected when the Slurm session ends, such as when it reaches its time limit.
Note that if you need to use a conda environment, you'll have to configure VS Code to use the Python executable you found above, since conda is not installed on the compute nodes.

=== SSH to Compute Nodes ===
The ability to SSH directly to compute nodes is disabled. You can run the command below to start a bash terminal on a compute node when running a batch job. This allows for the same functionality as SSH.<syntaxhighlight lang="bash">
srun —jobid=yourjobidhere --pty /bin/bash
</syntaxhighlight>

== Monitoring cluster usage ==
The [https://util.stat.cmu.edu/grafana/d/Oxed_c6Wz/nvidia-wright-gpu-metrics NVIDIA Wright GPU Metrics Dashboard] shows graphs of GPU usage across the cluster. At the top left, you can select which node to graph, so viewing your department's node will let you see how GPUs are being used.

If you're developing a long-running job, we recommend checking this dashboard while your job is running. It is easy to accidentally write PyTorch or Tensorflow code that '''doesn't actually use the GPU''', so check the graphs to see if the expected number of GPUs are in used based on your job's requirements and the other jobs listed in <code>squeue</code>. If they are not, get a terminal on the compute node as described above and run <code>nvidia-smi</code> to get GPU usage information, including which scripts are accessing the GPU. If your script doesn't show up, you can debug it to figure out why no GPUs are being used.

== Using scratch storage ==
Each compute node (but not the head node) has several terabytes of scratch space. This is fast SSD storage that is directly attached to the node, rather than being accessed over the network (like all files in <code>/home</code> are). This is useful if your jobs need to write large temporary files: reading and writing the scratch space is faster than reading and writing files in <code>/home</code>.

Your home directory contains a symlink to the scratch space at <code>~/scratch</code>. To use it, create a directory inside named after your username:
mkdir ~/scratch/yourusername
Then you can read and write files in that directory.

'''Warnings:'''

* The scratch space is meant for '''temporary''' storage during a job. It may be cleared out during cluster maintenance or if the disk is full.
* The scratch drives are much smaller than the huge networked disk used for <code>/home</code>, so be careful what you store in it, and regularly delete files you don't need.
* Each node has separate scratch disks, so data you write on one node is '''not''' available from any other node, or from the head node.

The scratch space is thus not for final model outputs or results. It is ideal for things like saving checkpoints while your job is running, dumping data to debug a job, saving output from one script to be immediately used by another script, etc.

Wright GPU Cluster

2026-01-27T16:07:43Z

Areinhar: Describe scratch space

The department has access to a powerful GPU-based cluster named [https://en.wikipedia.org/wiki/Sewall_Wright Wright] which is hosted here at CMU. This is a 4-node and growing cluster that is a shared resource for Statistics & Data Science, Philosophy, and other departments in Dietrich College. Each department has priority access to its own compute node, but will be able to share the resources from other departments when not in use.

== Access requests ==
This resource is free to use to users in the Departments of Statistics & Data Science, Philosophy, and CMIST. These departments set policies determining which of their users are eligible. Users outside these departments must be sponsored by someone in the eligible departments, as part of a project or collaboration.

To request access, complete the [https://docs.google.com/forms/d/e/1FAIpQLSftCsplE06whGocn62DhwC6NpDSgZ-DMmUacrQID_slIfg6sA/viewform access request form] and our team will be in touch to discuss your request and onboard you.

== Cluster setup ==
The cluster currently consists of 4 servers:
*The head node: You log in to this to write code and run jobs. It has 128 GB of RAM, 16 cores, and '''no GPU'''.
*The compute nodes: Jobs run on these. Each has 256 CPU cores, 8 NVIDIA L40 GPU cards with 48 GB of video RAM each, approximately 1 TB of RAM, and a total of approximately 3 TB of local data/scratch space.
*The file server: Holds 50TB of storage. The head node and compute nodes connect via fiberoptic networking. Your home directory is hosted on the file server, so the same files are accessible on the head node and all compute nodes.
The cluster is currently partitioned. One compute node is reserved for each department involved.

The file server has quotas enabled so that each department gets 1/3rd of the available storage. If a department user tries to add files beyond the quota, the writes will be rejected, and nobody in that department will be able to add files until space is freed. To check the quotas and your own usage, [https://util.stat.cmu.edu/quotareport/ view the quota report].

== Connecting and setting up your environment ==

=== SSH ===

From your terminal, type <code>ssh <YourStatID>@wright.hss.cmu.edu</code> . You can also avoid remembering this by adding the following snippet to your local <code>~/.ssh/config</code> file:
<pre>Host wright
Hostname wright.hss.cmu.edu
User YourStatID
</pre>

You can then connect by simply typing <code>ssh wright</code>.

This connects you to the '''head node.''' Here you can upload your data files, set up your scripts, and test things.

'''Do not run resource-intensive tasks directly on the head node'''. It doesn't have any GPUs, and it has fewer resources than the compute nodes. You can do quick tests to make sure your scripts are set up correctly, but submit all other jobs through Slurm.

=== Uploading files ===
To upload scripts or data files, you can use <code>scp</code> from your local computer (Mac or Linux); on Windows, you can get the Tectia SSH graphical client [https://www.cmu.edu/computing/services/endpoint/software/access.html free through CMU].

You can also connect through the [https://code.visualstudio.com/docs/remote/ssh Remote SSH feature of VS Code] if you like. This is useful to 

*navigate the file system from a proper GUI and 
*modify code/files/folder in place if you are not familiar with the command line
Files uploaded to your home directory will be accessible on the head node and all compute nodes, so they can be freely used in compute jobs.

=== Using Conda ===
The head node has Anaconda installed for managing Python packages and system libraries; see [[Python virtual environments]] for information on how to use it (but note it's '''already installed''', so you don't need to install it from scratch!).

However, the compute nodes '''do not''' have conda, so your slurm jobs can't use <code>conda activate</code> or other familiar commands. Instead, set up your environment on the head node, activate it, install the packages you need, and so on. Then run <code>which python</code> to get the path to the Python executable in your environment. For instance:<syntaxhighlight lang="bash">
$ which python
/home/export/areinhar/.conda/envs/llm/bin/python
</syntaxhighlight>This is the path you'll use in your scripts (see below).

=== Open OnDemand ===

Open OnDemand is a graphical interface where you can run jobs, interactive applications, and manage files. More information can be found [[Open OnDemand Wright|here]].

== Job scheduling and prioritization ==

The GPU cluster manages jobs via [[Main_Page/slurm|Slurm]]. Slurm is a job management system that deals with job prioritization and the allocation of resources. On high-performance computers (e.g., PSC and our GPU cluster) Slurm helps allocate resources so multiple jobs can be run quickly. Users run programs by submitting scripts to the Slurm job scheduler, and the scheduler runs them when resources are available.
=== Queue status ===

To check what jobs are running, use <code>squeue</code>. The following options list more helpful information than the default (though you need a wide terminal to read it all):<syntaxhighlight lang="bash">
$ squeue -O "JobID:7,UserName:10,Partition:12,TimeUsed:12,ReasonList,MinCPUS:9,MinMemory:12,tres-per-node,StartTime,EndTime"
JOBID USER PARTITION TIME NODELIST(REASON) MIN_CPUS MIN_MEMORY TRES_PER_NODE START_TIME END_TIME
36229 ebensen all 0:00 (DependencyNeverSati16 128G N/A N/A N/A
36237 ajshen all 12:35:57 n02 2 64G gres:gpu:1 2026-01-26T21:55:23 2026-01-28T21:55:23
36152 msezgin all 1-02:32:04 n03 1 64G gres:gpu:1 2026-01-26T07:59:16 2026-01-28T07:59:16
36132 tsukiant cmist_condo 1-11:00:02 n03 16 8G gres:gpu:1 2026-01-25T23:31:18 2026-01-27T23:31:18
36100 tsukiant cmist_condo 1-17:32:39 n03 16 8G gres:gpu:1 2026-01-25T16:58:41 2026-01-27T16:58:41
36090 tsukiant cmist_condo 1-18:22:51 n03 16 8G gres:gpu:1 2026-01-25T16:08:29 2026-01-27T16:08:29
36247 jwalches statds 42:38 n01 40 15G gres:gpu:3 2026-01-27T09:48:42 2026-01-29T09:48:42
36051 xiangchs statds_condo7:05:24 n01 64 128G gres:gpu:4 2026-01-27T03:25:56 2026-01-29T03:25:56
</syntaxhighlight>

Here we see there are two jobs running on node 1, one on node 2, four on node 3, and one job that is not running because it was submitted with requirements that will not be met. The listed end times are based on the time limit specified when the job was submitted, so the jobs may end sooner. If you submit a job that is waiting for resources, the reason will be listed as <code>(Resources)</code> or <code>(Priority)</code>, and after a few minutes Slurm will list a start time for your job based on when other jobs will finish and free up resources.

To check what resources are currently in use, use <code>sinfo</code>:<syntaxhighlight lang="bash">
$ sinfo -O "NodeList:10,CPUsState:16,Memory:9,FreeMem:10,Gres:20,GresUsed:30"
NODELIST CPUS(A/I/O/T) MEMORY FREE_MEM GRES GRES_USED
n01 64/192/0/256 1000000 221943 gpu:nvidia-l40:8 gpu:nvidia-l40:8(IDX:0-7)
n02 32/224/0/256 1000000 225575 gpu:nvidia-l40:8 gpu:nvidia-l40:0(IDX:N/A)
</syntaxhighlight>This shows that node 1 (Statistics & Data Science) has 64 CPUs allocated and 192 idle, out of 256 total. There is 1TB of RAM, of which 222 GB is free. There are 8 NVIDIA L40 GPus, all 8 of which are in use. Hence if we submit a job to the statds partition requesting any GPUs, it will have to wait until some become free. Meanwhile, node 2 (Philosophy) has no GPUs in use.

=== Partitions and condos ===
The cluster is divided into three partitions:

* <code>statds</code> (Statistics & Data Science)
* <code>phil</code> (Philosophy)
* <code>cmist</code> (CMIST)

Jobs are submitted to a partition. '''Users must submit jobs to their department's partition''' -- jobs submitted to the wrong partition will be rejected.

There are several other special partitions:

* <code>all</code> requests any available node from any partition. Thus a Philosophy user can run a job on the CMIST node, or vice versa.
* <code>statds_condo</code>, <code>phil_condo</code>, and <code>cmist_condo</code> are "condo" partitions. Any user can submit to any of these partitions.

These special partitions are like holiday condos: any user can access them, but with a lower priority than the owner (the host department). Jobs running on them are also preemptible. For example, if a Philosophy user submits to <code>statds_condo</code>, their job will run only when Statistics users are not using the resources. If a Statistics user submits a large job after yours, your job might be terminated, and will automatically run again when resources become available. The <code>all</code> partition is similar, but for whichever node happens to be free.

=== Job prioritization ===
Slurm runs jobs based on the resources they request and what is available. If resources aren't available for your job, slurm will put it in the queue and wait until other jobs finish. However, it's not simply a first-in first-out queue -- Slurm attempts to arrange the jobs so every job can run, even if that means running them out of order. Slurm also adjusts each user's priority based on how many jobs they submit, so that one user can't hog the cluster by submitting dozens of jobs; if one user has many jobs waiting, theirs will be reduced in priority so other users can run jobs as well.

Examples:

* Someone submits Job A that requests 6 GPUs. Later, you submit Job B that requests 4 GPUs on the same partition. The compute node only has 8 GPUs, so Job B will be queued until Job A completes.
* Someone submits Job A that requests 6 GPUs. Later, you submit Job B that requests 2 GPUs. Job B can run immediately.
* You submit a job that requires all 8 GPUs on a node, but 4 are currently in use. Your job is put in the queue and scheduled to run when Slurm knows the current jobs will finish (because of their maximum time limit). Meanwhile, someone submits a job using 2 GPUs and a short time limit. Slurm lets it run immediately, because it knows it will be done before your job is scheduled to run. Other jobs may be put off until after yours runs, so yours can get the 8 GPUs it needs.
* You submit 50 jobs that each require 1 GPU. The node has only 8 GPUs, but several are in use. Several of your jobs run, but the others are queued for later. Other users can submit jobs while yours are in the queue, and some of their jobs may run before some of yours, for fairness.
* You submit a job to a condo partition. The partition is currently full, so your job is queued. Other users from that department submit new jobs. As they are higher priority than you, their jobs run first.
You can view the currently running or queued jobs by running <code>squeue</code> on the head node, as described above. The <code>NODELIST(REASON)</code> column will indicate which jobs are queued by giving special codes:

* <code>(Resources)</code> means there are not enough resources (CPUs, memory, or GPUs) available for the job. It will run once the resources are free.
* <code>(Priority)</code> means resources are available, but Slurm wants to run a higher-priority job first. For example, maybe a large job has been waiting for resources for two days before you submit your job; Slurm may make your job wait so the large job can start first.

=== Job etiquette ===
The cluster is shared between users and departments. Therefore:

* Request the minimum resources required for your job, so other users can access other resources.
* If you need to submit a large number of jobs and aren't in a hurry, consider using your department's condo partition or <code>all</code>. Using <code>all</code> lets you get all available resources, but at lower priority, so other users can run jobs as well and you don't hog the cluster.
* Watch your disk space usage. With large deep models it is easy to use many terabytes of storage, and each department has a quota. [https://util.stat.cmu.edu/quotareport/ You can check the quota and your own usage here.]

== Setting up jobs ==

=== Submitting batch jobs ===
The best way to run jobs is by writing a shell script specifying the Slurm options. To launch a batch job from a script you will need two files:

*the actual script in your favorite programming language that does whatever computation you want
*a bash script that requests the resources you need from Slurm and gives the commands to launch your job.
A typical bash script might look like this:<pre>#!/bin/bash
# specify partition to run in (statds, phil, or cmist)
#SBATCH --partition statds
#SBATCH --ntasks 1
#SBATCH --job-name=nameofyourjob

# number of CPUs you need:
#SBATCH --cpus-per-task 1

# amount of memory per CPU
# (total allocated is CPUs * memory)
#SBATCH --mem-per-cpu 4G

# number of GPUs you need
#SBATCH --gres=gpu:1

# maximum time limit for job, HH:MM:SS (up to 48 hours max, 48:00:00)
#SBATCH --time=01:00:00

# email you when job is finished or errors out:
#SBATCH --mail-user=YourAndrewID@andrew.cmu.edu
#SBATCH --mail-type=END,FAIL

## Choose one:

# to run a Python script in a Conda environment called ENV
srun /home/export/YOURANDREWID/.conda/envs/ENV/bin/python YourScript.py

# to run an R script:
srun Rscript YourScript.R
</pre>Adjust the number of CPUs, number of GPUs, memory usage, and time limit based on the needs of your script. Requesting more than you need may result in your job having to wait for resources to free up; requesting a long time you don't need will block slurm from scheduling other jobs for when yours will be done. (And if you request more resources than exist on the compute nodes, your job will be queued forever while it waits for resources to appear.)

'''Note:''' Slurm batch jobs are limited to a maximum duration of 48 hours. Jobs running longer will be killed.

To run the job, SSH into the head node and navigate to the directory with your scripts. Then run:<syntaxhighlight lang="bash">
sbatch example.sh
</syntaxhighlight>where <code>example.sh</code> is the name of your bash script.

You will see a message like "Submitted batch job 1209" when your job is submitted. Any printed output from your script will automatically be redirected to a file based on the job number -- in this case, <code>slurm-1209.out</code>. You can use <code>tail -f slurm-1209.out</code> to view the output live and monitor your script's progress.

=== Selecting the right resources ===
In the Slurm script above, you can choose the number of tasks, the number of GPUs, the memory per CPU, the number of GPUs, and the time limit. Choose these values wisely:

* You generally want '''1 task''' (<code>--ntasks 1</code>). Asking for multiple tasks will cause srun to run multiple copies of your script simultaneously. If you need to run the same script with 10 different parameter values, this can be useful: request 10 tasks, and read the <code>SLURM_PROCID</code> environment variable in your script. It will be numbered 0-9 (for 10 tasks), so you can use this to set the parameter values for each parallel run.
* Set the number of CPUs based on your script. If your script is entirely single-threaded, ask for 1. If you use parallel processing (such as with multiprocessing in Python or the parallel package in R), use a multithreaded package, or spawn many parallel programs in your job, request more.
* Request the minimum amount of memory (RAM) needed for your script. This is per-CPU, so requesting a lot of CPUs and RAM can easily mean you request more RAM than exists on the node.
* Start with 1 GPU, adding more if your job requires more GPU memory (VRAM) than is on one GPU, or if you explicitly write your script to use more. If you do not use GPUs, request 0 so you aren't allocated a GPU that nobody else can use while your job runs.
* Set the time limit a little longer than you expect your script to run. Slurm uses the time limit when scheduling, so always choosing the maximum (48 hours) will backfire: Slurm will assume your job will take a long time and will run other things first. If you know your job can only take an hour, setting, say, a 01:30:00 time limit will let Slurm fit your job in before other long jobs in the queue. But of course if you choose too low and your job exceeds the limit, it will be killed, so leave yourself some extra time.

=== Interactive mode ===

You can also run interactive sessions that allow you to, say, run an R or Python console and type commands and immediately view the output. From the head node, run a command like:<syntaxhighlight lang="bash">
# remember to edit to correct partition:
srun --partition statds --ntasks 1 \
--cpus-per-task 16 --mem-per-cpu 4G \
--gres=gpu:1 --time=01:00:00 --pty \
bash
</syntaxhighlight>This runs a job that opens a bash shell on the compute node, and connects you directly to this job. You can now run bash commands, open Python, or run R scripts. Note your interactive session has a time limit and will be killed when it elapses, so set the time limit carefully and choose the resource amounts in the command according to your needs.

Interactive sessions are limited to '''two hours maximum'''. You cannot request a longer time limit. This is because interactive sessions are inefficient, with the cluster spending most of its time waiting for you to type something instead of running code. For large tasks, submit a batch job.

=== Interactive via VS Code ===
VS Code supports SSH and tunneling, which allows it to connect to a server and run commands. This allows you to use the Jupyter Notebook support built into VS Code to interact with the GPU cluster. As this uses an interactive session, it's subject to the same '''two-hour time limit'''. Hence notebooks should be used only for exploration and debugging, not large jobs.

To use this, we need to connect to the head node, run an interactive Slurm session, and then connect VS Code to that Slurm session so it can use it. This takes a few extra steps.

==== Setup ====
Before you use this for the first time, do the following steps:

*SSH into Wright and run the following to install and unpack the VSCode CLI: 
<pre>curl -Lk 'https://code.visualstudio.com/sha/download?build=stable&os=cli-alpine-x64' --output vscode_cli.tar.gz
tar -xf vscode_cli.tar.gz
</pre>

*Add the <code>code</code> executable to your PATH by adding the following to your <code>.bashrc</code> file: <code>export PATH="~:$PATH"</code><pre>echo 'export PATH="$PATH:/path/to/code"' >> ~/.bashrc
</pre>
*In VSCode, install the <code>Remote - Tunnels</code> extension, then use the VSCode SSH feature to connect to Wright and make sure to install all the local VS Code extensions on the remote machine as well (e.g., Python, Jupyter, ...)

==== Running a session ====
*In the Wright headnode, launch a Slurm interactive session with <code>srun</code>, following the instructions above.

*Inside the interactive session, run <code>code tunnel</code> and follow the instructions. The first time the system will ask you to name the tunnel and authorize the process via an authenticator, I suggest using GitHub since it usually works quite smoothly. Subsequent tunnels might still ask you to authorize their creation by typing an alphanumerical sequence into a link that appears in the terminal. Once you’ve done that, the tunnel is successfully created.
*From VSCode, select <code>Connect to Tunnel</code>, then choose the authentication method you previously used and you will see the tunnel appear in the dropdown. Select it and you will land directly into the interactive session we just created
*You will now be able to run code and, e.g., Jupyter Notebooks using the compute node allocated by the Slurm interactive session. VS Code will be disconnected when the Slurm session ends, such as when it reaches its time limit.
Note that if you need to use a conda environment, you'll have to configure VS Code to use the Python executable you found above, since conda is not installed on the compute nodes.

=== SSH to Compute Nodes ===
The ability to SSH directly to compute nodes is disabled. You can run the command below to start a bash terminal on a compute node when running a batch job. This allows for the same functionality as SSH.<syntaxhighlight lang="bash">
srun —jobid=yourjobidhere --pty /bin/bash
</syntaxhighlight>

== Monitoring cluster usage ==
The [https://util.stat.cmu.edu/grafana/d/Oxed_c6Wz/nvidia-wright-gpu-metrics NVIDIA Wright GPU Metrics Dashboard] shows graphs of GPU usage across the cluster. At the top left, you can select which node to graph, so viewing your department's node will let you see how GPUs are being used.

If you're developing a long-running job, we recommend checking this dashboard while your job is running. It is easy to accidentally write PyTorch or Tensorflow code that '''doesn't actually use the GPU''', so check the graphs to see if the expected number of GPUs are in used based on your job's requirements and the other jobs listed in <code>squeue</code>. If they are not, get a terminal on the compute node as described above and run <code>nvidia-smi</code> to get GPU usage information, including which scripts are accessing the GPU. If your script doesn't show up, you can debug it to figure out why no GPUs are being used.

== Using scratch storage ==
Each compute node (but not the head node) has several terabytes of scratch space. This is fast SSD storage that is directly attached to the node, rather than being accessed over the network (like all files in <code>/home</code> are). This is useful if your jobs need to write large temporary files: reading and writing the scratch space is faster than reading and writing files in <code>/home</code>.

Your home directory contains a symlink to the scratch space at <code>~/scratch</code>. To use it, create a directory inside named after your username:
mkdir ~/scratch/yourusername
Then you can read and write files in that directory.

'''Warnings:'''

* The scratch space is meant for '''temporary''' storage during a job. It may be cleared out during cluster maintenance or if the disk is full.
* The scratch drives are much smaller than the huge networked disk used for <code>/home</code>, so be careful what you store in it, and regularly delete files you don't need.
* Each node has separate scratch disks, so data you write on one node is '''not''' available from any other node, or from the head node.

The scratch space is thus not for final model outputs or results. It is ideal for things like saving checkpoints while your job is running, dumping data to debug a job, saving output from one script to be immediately used by another script, etc.

Wright GPU Cluster

2026-01-27T15:38:16Z

Areinhar: /* Queue status */

The department has access to a powerful GPU-based cluster named [https://en.wikipedia.org/wiki/Sewall_Wright Wright] which is hosted here at CMU. This is a 4-node and growing cluster that is a shared resource for Statistics & Data Science, Philosophy, and other departments in Dietrich College. Each department has priority access to its own compute node, but will be able to share the resources from other departments when not in use.

== Access requests ==
This resource is free to use to users in the Departments of Statistics & Data Science, Philosophy, and CMIST. These departments set policies determining which of their users are eligible. Users outside these departments must be sponsored by someone in the eligible departments, as part of a project or collaboration.

To request access, complete the [https://docs.google.com/forms/d/e/1FAIpQLSftCsplE06whGocn62DhwC6NpDSgZ-DMmUacrQID_slIfg6sA/viewform access request form] and our team will be in touch to discuss your request and onboard you.

== Cluster setup ==
The cluster currently consists of 4 servers:
*The head node: You log in to this to write code and run jobs. It has 128 GB of RAM, 16 cores, and '''no GPU'''.
*The compute nodes: Jobs run on these. Each has 256 CPU cores, 8 NVIDIA L40 GPU cards with 48 GB of video RAM each, approximately 1 TB of RAM, and a total of approximately 3 TB of local data/scratch space.
*The file server: Holds 50TB of storage. The head node and compute nodes connect via fiberoptic networking. Your home directory is hosted on the file server, so the same files are accessible on the head node and all compute nodes.
The cluster is currently partitioned. One compute node is reserved for each department involved.

The file server has quotas enabled so that each department gets 1/3rd of the available storage. If a department user tries to add files beyond the quota, the writes will be rejected, and nobody in that department will be able to add files until space is freed. To check the quotas and your own usage, [https://util.stat.cmu.edu/quotareport/ view the quota report].

== Connecting and setting up your environment ==

=== SSH ===

From your terminal, type <code>ssh <YourStatID>@wright.hss.cmu.edu</code> . You can also avoid remembering this by adding the following snippet to your local <code>~/.ssh/config</code> file:
<pre>Host wright
Hostname wright.hss.cmu.edu
User YourStatID
</pre>

You can then connect by simply typing <code>ssh wright</code>.

This connects you to the '''head node.''' Here you can upload your data files, set up your scripts, and test things.

'''Do not run resource-intensive tasks directly on the head node'''. It doesn't have any GPUs, and it has fewer resources than the compute nodes. You can do quick tests to make sure your scripts are set up correctly, but submit all other jobs through Slurm.

=== Uploading files ===
To upload scripts or data files, you can use <code>scp</code> from your local computer (Mac or Linux); on Windows, you can get the Tectia SSH graphical client [https://www.cmu.edu/computing/services/endpoint/software/access.html free through CMU].

You can also connect through the [https://code.visualstudio.com/docs/remote/ssh Remote SSH feature of VS Code] if you like. This is useful to 

*navigate the file system from a proper GUI and 
*modify code/files/folder in place if you are not familiar with the command line
Files uploaded to your home directory will be accessible on the head node and all compute nodes, so they can be freely used in compute jobs.

=== Using Conda ===
The head node has Anaconda installed for managing Python packages and system libraries; see [[Python virtual environments]] for information on how to use it (but note it's '''already installed''', so you don't need to install it from scratch!).

However, the compute nodes '''do not''' have conda, so your slurm jobs can't use <code>conda activate</code> or other familiar commands. Instead, set up your environment on the head node, activate it, install the packages you need, and so on. Then run <code>which python</code> to get the path to the Python executable in your environment. For instance:<syntaxhighlight lang="bash">
$ which python
/home/export/areinhar/.conda/envs/llm/bin/python
</syntaxhighlight>This is the path you'll use in your scripts (see below).

=== Open OnDemand ===

Open OnDemand is a graphical interface where you can run jobs, interactive applications, and manage files. More information can be found [[Open OnDemand Wright|here]].

== Job scheduling and prioritization ==

The GPU cluster manages jobs via [[Main_Page/slurm|Slurm]]. Slurm is a job management system that deals with job prioritization and the allocation of resources. On high-performance computers (e.g., PSC and our GPU cluster) Slurm helps allocate resources so multiple jobs can be run quickly. Users run programs by submitting scripts to the Slurm job scheduler, and the scheduler runs them when resources are available.
=== Queue status ===

To check what jobs are running, use <code>squeue</code>. The following options list more helpful information than the default (though you need a wide terminal to read it all):<syntaxhighlight lang="bash">
$ squeue -O "JobID:7,UserName:10,Partition:12,TimeUsed:12,ReasonList,MinCPUS:9,MinMemory:12,tres-per-node,StartTime,EndTime"
JOBID USER PARTITION TIME NODELIST(REASON) MIN_CPUS MIN_MEMORY TRES_PER_NODE START_TIME END_TIME
36229 ebensen all 0:00 (DependencyNeverSati16 128G N/A N/A N/A
36237 ajshen all 12:35:57 n02 2 64G gres:gpu:1 2026-01-26T21:55:23 2026-01-28T21:55:23
36152 msezgin all 1-02:32:04 n03 1 64G gres:gpu:1 2026-01-26T07:59:16 2026-01-28T07:59:16
36132 tsukiant cmist_condo 1-11:00:02 n03 16 8G gres:gpu:1 2026-01-25T23:31:18 2026-01-27T23:31:18
36100 tsukiant cmist_condo 1-17:32:39 n03 16 8G gres:gpu:1 2026-01-25T16:58:41 2026-01-27T16:58:41
36090 tsukiant cmist_condo 1-18:22:51 n03 16 8G gres:gpu:1 2026-01-25T16:08:29 2026-01-27T16:08:29
36247 jwalches statds 42:38 n01 40 15G gres:gpu:3 2026-01-27T09:48:42 2026-01-29T09:48:42
36051 xiangchs statds_condo7:05:24 n01 64 128G gres:gpu:4 2026-01-27T03:25:56 2026-01-29T03:25:56
</syntaxhighlight>

Here we see there are two jobs running on node 1, one on node 2, four on node 3, and one job that is not running because it was submitted with requirements that will not be met. The listed end times are based on the time limit specified when the job was submitted, so the jobs may end sooner. If you submit a job that is waiting for resources, the reason will be listed as <code>(Resources)</code> or <code>(Priority)</code>, and after a few minutes Slurm will list a start time for your job based on when other jobs will finish and free up resources.

To check what resources are currently in use, use <code>sinfo</code>:<syntaxhighlight lang="bash">
$ sinfo -O "NodeList:10,CPUsState:16,Memory:9,FreeMem:10,Gres:20,GresUsed:30"
NODELIST CPUS(A/I/O/T) MEMORY FREE_MEM GRES GRES_USED
n01 64/192/0/256 1000000 221943 gpu:nvidia-l40:8 gpu:nvidia-l40:8(IDX:0-7)
n02 32/224/0/256 1000000 225575 gpu:nvidia-l40:8 gpu:nvidia-l40:0(IDX:N/A)
</syntaxhighlight>This shows that node 1 (Statistics & Data Science) has 64 CPUs allocated and 192 idle, out of 256 total. There is 1TB of RAM, of which 222 GB is free. There are 8 NVIDIA L40 GPus, all 8 of which are in use. Hence if we submit a job to the statds partition requesting any GPUs, it will have to wait until some become free. Meanwhile, node 2 (Philosophy) has no GPUs in use.

=== Partitions and condos ===
The cluster is divided into three partitions:

* <code>statds</code> (Statistics & Data Science)
* <code>phil</code> (Philosophy)
* <code>cmist</code> (CMIST)

Jobs are submitted to a partition. '''Users must submit jobs to their department's partition''' -- jobs submitted to the wrong partition will be rejected.

There are several other special partitions:

* <code>all</code> requests any available node from any partition. Thus a Philosophy user can run a job on the CMIST node, or vice versa.
* <code>statds_condo</code>, <code>phil_condo</code>, and <code>cmist_condo</code> are "condo" partitions. Any user can submit to any of these partitions.

These special partitions are like holiday condos: any user can access them, but with a lower priority than the owner (the host department). Jobs running on them are also preemptible. For example, if a Philosophy user submits to <code>statds_condo</code>, their job will run only when Statistics users are not using the resources. If a Statistics user submits a large job after yours, your job might be terminated, and will automatically run again when resources become available. The <code>all</code> partition is similar, but for whichever node happens to be free.

=== Job prioritization ===
Slurm runs jobs based on the resources they request and what is available. If resources aren't available for your job, slurm will put it in the queue and wait until other jobs finish. However, it's not simply a first-in first-out queue -- Slurm attempts to arrange the jobs so every job can run, even if that means running them out of order. Slurm also adjusts each user's priority based on how many jobs they submit, so that one user can't hog the cluster by submitting dozens of jobs; if one user has many jobs waiting, theirs will be reduced in priority so other users can run jobs as well.

Examples:

* Someone submits Job A that requests 6 GPUs. Later, you submit Job B that requests 4 GPUs on the same partition. The compute node only has 8 GPUs, so Job B will be queued until Job A completes.
* Someone submits Job A that requests 6 GPUs. Later, you submit Job B that requests 2 GPUs. Job B can run immediately.
* You submit a job that requires all 8 GPUs on a node, but 4 are currently in use. Your job is put in the queue and scheduled to run when Slurm knows the current jobs will finish (because of their maximum time limit). Meanwhile, someone submits a job using 2 GPUs and a short time limit. Slurm lets it run immediately, because it knows it will be done before your job is scheduled to run. Other jobs may be put off until after yours runs, so yours can get the 8 GPUs it needs.
* You submit 50 jobs that each require 1 GPU. The node has only 8 GPUs, but several are in use. Several of your jobs run, but the others are queued for later. Other users can submit jobs while yours are in the queue, and some of their jobs may run before some of yours, for fairness.
* You submit a job to a condo partition. The partition is currently full, so your job is queued. Other users from that department submit new jobs. As they are higher priority than you, their jobs run first.
You can view the currently running or queued jobs by running <code>squeue</code> on the head node, as described above. The <code>NODELIST(REASON)</code> column will indicate which jobs are queued by giving special codes:

* <code>(Resources)</code> means there are not enough resources (CPUs, memory, or GPUs) available for the job. It will run once the resources are free.
* <code>(Priority)</code> means resources are available, but Slurm wants to run a higher-priority job first. For example, maybe a large job has been waiting for resources for two days before you submit your job; Slurm may make your job wait so the large job can start first.

=== Job etiquette ===
The cluster is shared between users and departments. Therefore:

* Request the minimum resources required for your job, so other users can access other resources.
* If you need to submit a large number of jobs and aren't in a hurry, consider using your department's condo partition or <code>all</code>. Using <code>all</code> lets you get all available resources, but at lower priority, so other users can run jobs as well and you don't hog the cluster.
* Watch your disk space usage. With large deep models it is easy to use many terabytes of storage, and each department has a quota. [https://util.stat.cmu.edu/quotareport/ You can check the quota and your own usage here.]

== Setting up jobs ==

=== Submitting batch jobs ===
The best way to run jobs is by writing a shell script specifying the Slurm options. To launch a batch job from a script you will need two files:

*the actual script in your favorite programming language that does whatever computation you want
*a bash script that requests the resources you need from Slurm and gives the commands to launch your job.
A typical bash script might look like this:<pre>#!/bin/bash
# specify partition to run in (statds, phil, or cmist)
#SBATCH --partition statds
#SBATCH --ntasks 1
#SBATCH --job-name=nameofyourjob

# number of CPUs you need:
#SBATCH --cpus-per-task 1

# amount of memory per CPU
# (total allocated is CPUs * memory)
#SBATCH --mem-per-cpu 4G

# number of GPUs you need
#SBATCH --gres=gpu:1

# maximum time limit for job, HH:MM:SS (up to 48 hours max, 48:00:00)
#SBATCH --time=01:00:00

# email you when job is finished or errors out:
#SBATCH --mail-user=YourAndrewID@andrew.cmu.edu
#SBATCH --mail-type=END,FAIL

## Choose one:

# to run a Python script in a Conda environment called ENV
srun /home/export/YOURANDREWID/.conda/envs/ENV/bin/python YourScript.py

# to run an R script:
srun Rscript YourScript.R
</pre>Adjust the number of CPUs, number of GPUs, memory usage, and time limit based on the needs of your script. Requesting more than you need may result in your job having to wait for resources to free up; requesting a long time you don't need will block slurm from scheduling other jobs for when yours will be done. (And if you request more resources than exist on the compute nodes, your job will be queued forever while it waits for resources to appear.)

'''Note:''' Slurm batch jobs are limited to a maximum duration of 48 hours. Jobs running longer will be killed.

To run the job, SSH into the head node and navigate to the directory with your scripts. Then run:<syntaxhighlight lang="bash">
sbatch example.sh
</syntaxhighlight>where <code>example.sh</code> is the name of your bash script.

You will see a message like "Submitted batch job 1209" when your job is submitted. Any printed output from your script will automatically be redirected to a file based on the job number -- in this case, <code>slurm-1209.out</code>. You can use <code>tail -f slurm-1209.out</code> to view the output live and monitor your script's progress.

=== Selecting the right resources ===
In the Slurm script above, you can choose the number of tasks, the number of GPUs, the memory per CPU, the number of GPUs, and the time limit. Choose these values wisely:

* You generally want '''1 task''' (<code>--ntasks 1</code>). Asking for multiple tasks will cause srun to run multiple copies of your script simultaneously. If you need to run the same script with 10 different parameter values, this can be useful: request 10 tasks, and read the <code>SLURM_PROCID</code> environment variable in your script. It will be numbered 0-9 (for 10 tasks), so you can use this to set the parameter values for each parallel run.
* Set the number of CPUs based on your script. If your script is entirely single-threaded, ask for 1. If you use parallel processing (such as with multiprocessing in Python or the parallel package in R), use a multithreaded package, or spawn many parallel programs in your job, request more.
* Request the minimum amount of memory (RAM) needed for your script. This is per-CPU, so requesting a lot of CPUs and RAM can easily mean you request more RAM than exists on the node.
* Start with 1 GPU, adding more if your job requires more GPU memory (VRAM) than is on one GPU, or if you explicitly write your script to use more. If you do not use GPUs, request 0 so you aren't allocated a GPU that nobody else can use while your job runs.
* Set the time limit a little longer than you expect your script to run. Slurm uses the time limit when scheduling, so always choosing the maximum (48 hours) will backfire: Slurm will assume your job will take a long time and will run other things first. If you know your job can only take an hour, setting, say, a 01:30:00 time limit will let Slurm fit your job in before other long jobs in the queue. But of course if you choose too low and your job exceeds the limit, it will be killed, so leave yourself some extra time.

=== Interactive mode ===

You can also run interactive sessions that allow you to, say, run an R or Python console and type commands and immediately view the output. From the head node, run a command like:<syntaxhighlight lang="bash">
# remember to edit to correct partition:
srun --partition statds --ntasks 1 \
--cpus-per-task 16 --mem-per-cpu 4G \
--gres=gpu:1 --time=01:00:00 --pty \
bash
</syntaxhighlight>This runs a job that opens a bash shell on the compute node, and connects you directly to this job. You can now run bash commands, open Python, or run R scripts. Note your interactive session has a time limit and will be killed when it elapses, so set the time limit carefully and choose the resource amounts in the command according to your needs.

Interactive sessions are limited to '''two hours maximum'''. You cannot request a longer time limit. This is because interactive sessions are inefficient, with the cluster spending most of its time waiting for you to type something instead of running code. For large tasks, submit a batch job.

=== Interactive via VS Code ===
VS Code supports SSH and tunneling, which allows it to connect to a server and run commands. This allows you to use the Jupyter Notebook support built into VS Code to interact with the GPU cluster. As this uses an interactive session, it's subject to the same '''two-hour time limit'''. Hence notebooks should be used only for exploration and debugging, not large jobs.

To use this, we need to connect to the head node, run an interactive Slurm session, and then connect VS Code to that Slurm session so it can use it. This takes a few extra steps.

==== Setup ====
Before you use this for the first time, do the following steps:

*SSH into Wright and run the following to install and unpack the VSCode CLI: 
<pre>curl -Lk 'https://code.visualstudio.com/sha/download?build=stable&os=cli-alpine-x64' --output vscode_cli.tar.gz
tar -xf vscode_cli.tar.gz
</pre>

*Add the <code>code</code> executable to your PATH by adding the following to your <code>.bashrc</code> file: <code>export PATH="~:$PATH"</code><pre>echo 'export PATH="$PATH:/path/to/code"' >> ~/.bashrc
</pre>
*In VSCode, install the <code>Remote - Tunnels</code> extension, then use the VSCode SSH feature to connect to Wright and make sure to install all the local VS Code extensions on the remote machine as well (e.g., Python, Jupyter, ...)

==== Running a session ====
*In the Wright headnode, launch a Slurm interactive session with <code>srun</code>, following the instructions above.

*Inside the interactive session, run <code>code tunnel</code> and follow the instructions. The first time the system will ask you to name the tunnel and authorize the process via an authenticator, I suggest using GitHub since it usually works quite smoothly. Subsequent tunnels might still ask you to authorize their creation by typing an alphanumerical sequence into a link that appears in the terminal. Once you’ve done that, the tunnel is successfully created.
*From VSCode, select <code>Connect to Tunnel</code>, then choose the authentication method you previously used and you will see the tunnel appear in the dropdown. Select it and you will land directly into the interactive session we just created
*You will now be able to run code and, e.g., Jupyter Notebooks using the compute node allocated by the Slurm interactive session. VS Code will be disconnected when the Slurm session ends, such as when it reaches its time limit.
Note that if you need to use a conda environment, you'll have to configure VS Code to use the Python executable you found above, since conda is not installed on the compute nodes.

=== SSH to Compute Nodes ===
The ability to SSH directly to compute nodes is disabled. You can run the command below to start a bash terminal on a compute node when running a batch job. This allows for the same functionality as SSH.<syntaxhighlight lang="bash">
srun —jobid=yourjobidhere --pty /bin/bash
</syntaxhighlight>

== Monitoring cluster usage ==
The [https://util.stat.cmu.edu/grafana/d/Oxed_c6Wz/nvidia-wright-gpu-metrics NVIDIA Wright GPU Metrics Dashboard] shows graphs of GPU usage across the cluster. At the top left, you can select which node to graph, so viewing your department's node will let you see how GPUs are being used.

If you're developing a long-running job, we recommend checking this dashboard while your job is running. It is easy to accidentally write PyTorch or Tensorflow code that '''doesn't actually use the GPU''', so check the graphs to see if the expected number of GPUs are in used based on your job's requirements and the other jobs listed in <code>squeue</code>. If they are not, get a terminal on the compute node as described above and run <code>nvidia-smi</code> to get GPU usage information, including which scripts are accessing the GPU. If your script doesn't show up, you can debug it to figure out why no GPUs are being used.

Wright GPU Cluster

2026-01-20T17:57:47Z

Areinhar: job etiquette, clarify condos

The department has access to a powerful GPU-based cluster named [https://en.wikipedia.org/wiki/Sewall_Wright Wright] which is hosted here at CMU. This is a 4-node and growing cluster that is a shared resource for Statistics & Data Science, Philosophy, and other departments in Dietrich College. Each department has priority access to its own compute node, but will be able to share the resources from other departments when not in use.

== Access requests ==
This resource is free to use to users in the Departments of Statistics & Data Science, Philosophy, and CMIST. These departments set policies determining which of their users are eligible. Users outside these departments must be sponsored by someone in the eligible departments, as part of a project or collaboration.

To request access, complete the [https://docs.google.com/forms/d/e/1FAIpQLSftCsplE06whGocn62DhwC6NpDSgZ-DMmUacrQID_slIfg6sA/viewform access request form] and our team will be in touch to discuss your request and onboard you.

== Cluster setup ==
The cluster currently consists of 4 servers:
*The head node: You log in to this to write code and run jobs. It has 128 GB of RAM, 16 cores, and '''no GPU'''.
*The compute nodes: Jobs run on these. Each has 256 CPU cores, 8 NVIDIA L40 GPU cards with 48 GB of video RAM each, approximately 1 TB of RAM, and a total of approximately 3 TB of local data/scratch space.
*The file server: Holds 50TB of storage. The head node and compute nodes connect via fiberoptic networking. Your home directory is hosted on the file server, so the same files are accessible on the head node and all compute nodes.
The cluster is currently partitioned. One compute node is reserved for each department involved.

The file server has quotas enabled so that each department gets 1/3rd of the available storage. If a department user tries to add files beyond the quota, the writes will be rejected, and nobody in that department will be able to add files until space is freed. To check the quotas and your own usage, [https://util.stat.cmu.edu/quotareport/ view the quota report].

== Connecting and setting up your environment ==

=== SSH ===

From your terminal, type <code>ssh <YourStatID>@wright.hss.cmu.edu</code> . You can also avoid remembering this by adding the following snippet to your local <code>~/.ssh/config</code> file:
<pre>Host wright
Hostname wright.hss.cmu.edu
User YourStatID
</pre>

You can then connect by simply typing <code>ssh wright</code>.

This connects you to the '''head node.''' Here you can upload your data files, set up your scripts, and test things.

'''Do not run resource-intensive tasks directly on the head node'''. It doesn't have any GPUs, and it has fewer resources than the compute nodes. You can do quick tests to make sure your scripts are set up correctly, but submit all other jobs through Slurm.

=== Uploading files ===
To upload scripts or data files, you can use <code>scp</code> from your local computer (Mac or Linux); on Windows, you can get the Tectia SSH graphical client [https://www.cmu.edu/computing/services/endpoint/software/access.html free through CMU].

You can also connect through the [https://code.visualstudio.com/docs/remote/ssh Remote SSH feature of VS Code] if you like. This is useful to 

*navigate the file system from a proper GUI and 
*modify code/files/folder in place if you are not familiar with the command line
Files uploaded to your home directory will be accessible on the head node and all compute nodes, so they can be freely used in compute jobs.

=== Using Conda ===
The head node has Anaconda installed for managing Python packages and system libraries; see [[Python virtual environments]] for information on how to use it (but note it's '''already installed''', so you don't need to install it from scratch!).

However, the compute nodes '''do not''' have conda, so your slurm jobs can't use <code>conda activate</code> or other familiar commands. Instead, set up your environment on the head node, activate it, install the packages you need, and so on. Then run <code>which python</code> to get the path to the Python executable in your environment. For instance:<syntaxhighlight lang="bash">
$ which python
/home/export/areinhar/.conda/envs/llm/bin/python
</syntaxhighlight>This is the path you'll use in your scripts (see below).

=== Open OnDemand ===

Open OnDemand is a graphical interface where you can run jobs, interactive applications, and manage files. More information can be found [[Open OnDemand Wright|here]].

== Job scheduling and prioritization ==

The GPU cluster manages jobs via [[Main_Page/slurm|Slurm]]. Slurm is a job management system that deals with job prioritization and the allocation of resources. On high-performance computers (e.g., PSC and our GPU cluster) Slurm helps allocate resources so multiple jobs can be run quickly. Users run programs by submitting scripts to the Slurm job scheduler, and the scheduler runs them when resources are available.
=== Queue status ===

To check what jobs are running, use squeue:<syntaxhighlight lang="bash">
$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
1326 phil bash shaoanxi R 22:41:00 1 n02
1327 phil bash shaoanxi R 22:39:42 1 n02
1486 statds run_coun soheuny R 1:12:53 1 n01
1488 statds run_coun soheuny R 1:06:00 1 n01
1489 statds run_coun soheuny R 1:00:53 1 n01
1490 statds run_coun soheuny R 1:00:07 1 n01
1491 statds run_coun soheuny R 24:02 1 n01
1492 statds run_coun soheuny R 6:15 1 n01
1493 statds run_coun soheuny R 3:16 1 n01
1494 statds run_coun soheuny R 0:52 1 n01
</syntaxhighlight>

Here we see there are eight jobs running on node 1 and 2 jobs running on node 2.

To check what resources are currently in use, use <code>sinfo</code>:<syntaxhighlight lang="bash">
$ sinfo -O "NodeList:10,CPUsState:16,Memory:9,FreeMem:10,Gres:20,GresUsed:30"
NODELIST CPUS(A/I/O/T) MEMORY FREE_MEM GRES GRES_USED
n01 64/192/0/256 1000000 221943 gpu:nvidia-l40:8 gpu:nvidia-l40:8(IDX:0-7)
n02 32/224/0/256 1000000 225575 gpu:nvidia-l40:8 gpu:nvidia-l40:0(IDX:N/A)
</syntaxhighlight>This shows that node 1 (Statistics & Data Science) has 64 CPUs allocated and 192 idle, out of 256 total. There is 1TB of RAM, of which 222 GB is free. There are 8 NVIDIA L40 GPus, all 8 of which are in use. Hence if we submit a job to the statds partition requesting any GPUs, it will have to wait until some become free. Meanwhile, node 2 (Philosophy) has no GPUs in use.

=== Partitions and condos ===
The cluster is divided into three partitions:

* <code>statds</code> (Statistics & Data Science)
* <code>phil</code> (Philosophy)
* <code>cmist</code> (CMIST)

Jobs are submitted to a partition. '''Users must submit jobs to their department's partition''' -- jobs submitted to the wrong partition will be rejected.

There are several other special partitions:

* <code>all</code> requests any available node from any partition. Thus a Philosophy user can run a job on the CMIST node, or vice versa.
* <code>statds_condo</code>, <code>phil_condo</code>, and <code>cmist_condo</code> are "condo" partitions. Any user can submit to any of these partitions.

These special partitions are like holiday condos: any user can access them, but with a lower priority than the owner (the host department). Jobs running on them are also preemptible. For example, if a Philosophy user submits to <code>statds_condo</code>, their job will run only when Statistics users are not using the resources. If a Statistics user submits a large job after yours, your job might be terminated, and will automatically run again when resources become available. The <code>all</code> partition is similar, but for whichever node happens to be free.

=== Job prioritization ===
Slurm runs jobs based on the resources they request and what is available. If resources aren't available for your job, slurm will put it in the queue and wait until other jobs finish. However, it's not simply a first-in first-out queue -- Slurm attempts to arrange the jobs so every job can run, even if that means running them out of order. Slurm also adjusts each user's priority based on how many jobs they submit, so that one user can't hog the cluster by submitting dozens of jobs; if one user has many jobs waiting, theirs will be reduced in priority so other users can run jobs as well.

Examples:

* Someone submits Job A that requests 6 GPUs. Later, you submit Job B that requests 4 GPUs on the same partition. The compute node only has 8 GPUs, so Job B will be queued until Job A completes.
* Someone submits Job A that requests 6 GPUs. Later, you submit Job B that requests 2 GPUs. Job B can run immediately.
* You submit a job that requires all 8 GPUs on a node, but 4 are currently in use. Your job is put in the queue and scheduled to run when Slurm knows the current jobs will finish (because of their maximum time limit). Meanwhile, someone submits a job using 2 GPUs and a short time limit. Slurm lets it run immediately, because it knows it will be done before your job is scheduled to run. Other jobs may be put off until after yours runs, so yours can get the 8 GPUs it needs.
* You submit 50 jobs that each require 1 GPU. The node has only 8 GPUs, but several are in use. Several of your jobs run, but the others are queued for later. Other users can submit jobs while yours are in the queue, and some of their jobs may run before some of yours, for fairness.
* You submit a job to a condo partition. The partition is currently full, so your job is queued. Other users from that department submit new jobs. As they are higher priority than you, their jobs run first.
You can view the currently running or queued jobs by running <code>squeue</code> on the head node, as described above. The <code>NODELIST(REASON)</code> column will indicate which jobs are queued by giving special codes:

* <code>(Resources)</code> means there are not enough resources (CPUs, memory, or GPUs) available for the job. It will run once the resources are free.
* <code>(Priority)</code> means resources are available, but Slurm wants to run a higher-priority job first. For example, maybe a large job has been waiting for resources for two days before you submit your job; Slurm may make your job wait so the large job can start first.

=== Job etiquette ===
The cluster is shared between users and departments. Therefore:

* Request the minimum resources required for your job, so other users can access other resources.
* If you need to submit a large number of jobs and aren't in a hurry, consider using your department's condo partition or <code>all</code>. Using <code>all</code> lets you get all available resources, but at lower priority, so other users can run jobs as well and you don't hog the cluster.
* Watch your disk space usage. With large deep models it is easy to use many terabytes of storage, and each department has a quota. [https://util.stat.cmu.edu/quotareport/ You can check the quota and your own usage here.]

== Setting up jobs ==

=== Submitting batch jobs ===
The best way to run jobs is by writing a shell script specifying the Slurm options. To launch a batch job from a script you will need two files:

*the actual script in your favorite programming language that does whatever computation you want
*a bash script that requests the resources you need from Slurm and gives the commands to launch your job.
A typical bash script might look like this:<pre>#!/bin/bash
# specify partition to run in (statds, phil, or cmist)
#SBATCH --partition statds
#SBATCH --ntasks 1
#SBATCH --job-name=nameofyourjob

# number of CPUs you need:
#SBATCH --cpus-per-task 1

# amount of memory per CPU
# (total allocated is CPUs * memory)
#SBATCH --mem-per-cpu 4G

# number of GPUs you need
#SBATCH --gres=gpu:1

# maximum time limit for job, HH:MM:SS (up to 48 hours max, 48:00:00)
#SBATCH --time=01:00:00

# email you when job is finished or errors out:
#SBATCH --mail-user=YourAndrewID@andrew.cmu.edu
#SBATCH --mail-type=END,FAIL

## Choose one:

# to run a Python script in a Conda environment called ENV
srun /home/export/YOURANDREWID/.conda/envs/ENV/bin/python YourScript.py

# to run an R script:
srun Rscript YourScript.R
</pre>Adjust the number of CPUs, number of GPUs, memory usage, and time limit based on the needs of your script. Requesting more than you need may result in your job having to wait for resources to free up; requesting a long time you don't need will block slurm from scheduling other jobs for when yours will be done. (And if you request more resources than exist on the compute nodes, your job will be queued forever while it waits for resources to appear.)

'''Note:''' Slurm batch jobs are limited to a maximum duration of 48 hours. Jobs running longer will be killed.

To run the job, SSH into the head node and navigate to the directory with your scripts. Then run:<syntaxhighlight lang="bash">
sbatch example.sh
</syntaxhighlight>where <code>example.sh</code> is the name of your bash script.

You will see a message like "Submitted batch job 1209" when your job is submitted. Any printed output from your script will automatically be redirected to a file based on the job number -- in this case, <code>slurm-1209.out</code>. You can use <code>tail -f slurm-1209.out</code> to view the output live and monitor your script's progress.

=== Selecting the right resources ===
In the Slurm script above, you can choose the number of tasks, the number of GPUs, the memory per CPU, the number of GPUs, and the time limit. Choose these values wisely:

* You generally want '''1 task''' (<code>--ntasks 1</code>). Asking for multiple tasks will cause srun to run multiple copies of your script simultaneously. If you need to run the same script with 10 different parameter values, this can be useful: request 10 tasks, and read the <code>SLURM_PROCID</code> environment variable in your script. It will be numbered 0-9 (for 10 tasks), so you can use this to set the parameter values for each parallel run.
* Set the number of CPUs based on your script. If your script is entirely single-threaded, ask for 1. If you use parallel processing (such as with multiprocessing in Python or the parallel package in R), use a multithreaded package, or spawn many parallel programs in your job, request more.
* Request the minimum amount of memory (RAM) needed for your script. This is per-CPU, so requesting a lot of CPUs and RAM can easily mean you request more RAM than exists on the node.
* Start with 1 GPU, adding more if your job requires more GPU memory (VRAM) than is on one GPU, or if you explicitly write your script to use more. If you do not use GPUs, request 0 so you aren't allocated a GPU that nobody else can use while your job runs.
* Set the time limit a little longer than you expect your script to run. Slurm uses the time limit when scheduling, so always choosing the maximum (48 hours) will backfire: Slurm will assume your job will take a long time and will run other things first. If you know your job can only take an hour, setting, say, a 01:30:00 time limit will let Slurm fit your job in before other long jobs in the queue. But of course if you choose too low and your job exceeds the limit, it will be killed, so leave yourself some extra time.

=== Interactive mode ===

You can also run interactive sessions that allow you to, say, run an R or Python console and type commands and immediately view the output. From the head node, run a command like:<syntaxhighlight lang="bash">
# remember to edit to correct partition:
srun --partition statds --ntasks 1 \
--cpus-per-task 16 --mem-per-cpu 4G \
--gres=gpu:1 --time=01:00:00 --pty \
bash
</syntaxhighlight>This runs a job that opens a bash shell on the compute node, and connects you directly to this job. You can now run bash commands, open Python, or run R scripts. Note your interactive session has a time limit and will be killed when it elapses, so set the time limit carefully and choose the resource amounts in the command according to your needs.

Interactive sessions are limited to '''two hours maximum'''. You cannot request a longer time limit. This is because interactive sessions are inefficient, with the cluster spending most of its time waiting for you to type something instead of running code. For large tasks, submit a batch job.

=== Interactive via VS Code ===
VS Code supports SSH and tunneling, which allows it to connect to a server and run commands. This allows you to use the Jupyter Notebook support built into VS Code to interact with the GPU cluster. As this uses an interactive session, it's subject to the same '''two-hour time limit'''. Hence notebooks should be used only for exploration and debugging, not large jobs.

To use this, we need to connect to the head node, run an interactive Slurm session, and then connect VS Code to that Slurm session so it can use it. This takes a few extra steps.

==== Setup ====
Before you use this for the first time, do the following steps:

*SSH into Wright and run the following to install and unpack the VSCode CLI: 
<pre>curl -Lk 'https://code.visualstudio.com/sha/download?build=stable&os=cli-alpine-x64' --output vscode_cli.tar.gz
tar -xf vscode_cli.tar.gz
</pre>

*Add the <code>code</code> executable to your PATH by adding the following to your <code>.bashrc</code> file: <code>export PATH="~:$PATH"</code><pre>echo 'export PATH="$PATH:/path/to/code"' >> ~/.bashrc
</pre>
*In VSCode, install the <code>Remote - Tunnels</code> extension, then use the VSCode SSH feature to connect to Wright and make sure to install all the local VS Code extensions on the remote machine as well (e.g., Python, Jupyter, ...)

==== Running a session ====
*In the Wright headnode, launch a Slurm interactive session with <code>srun</code>, following the instructions above.

*Inside the interactive session, run <code>code tunnel</code> and follow the instructions. The first time the system will ask you to name the tunnel and authorize the process via an authenticator, I suggest using GitHub since it usually works quite smoothly. Subsequent tunnels might still ask you to authorize their creation by typing an alphanumerical sequence into a link that appears in the terminal. Once you’ve done that, the tunnel is successfully created.
*From VSCode, select <code>Connect to Tunnel</code>, then choose the authentication method you previously used and you will see the tunnel appear in the dropdown. Select it and you will land directly into the interactive session we just created
*You will now be able to run code and, e.g., Jupyter Notebooks using the compute node allocated by the Slurm interactive session. VS Code will be disconnected when the Slurm session ends, such as when it reaches its time limit.
Note that if you need to use a conda environment, you'll have to configure VS Code to use the Python executable you found above, since conda is not installed on the compute nodes.

=== SSH to Compute Nodes ===
The ability to SSH directly to compute nodes is disabled. You can run the command below to start a bash terminal on a compute node when running a batch job. This allows for the same functionality as SSH.<syntaxhighlight lang="bash">
srun —jobid=yourjobidhere --pty /bin/bash
</syntaxhighlight>

Wright GPU Cluster

2026-01-20T16:59:13Z

Areinhar: /* Submitting batch jobs */

The department has access to a powerful GPU-based cluster named [https://en.wikipedia.org/wiki/Sewall_Wright Wright] which is hosted here at CMU. This is a 4-node and growing cluster that is a shared resource for Statistics & Data Science, Philosophy, and other departments in Dietrich College. Each department has priority access to its own compute node, but will be able to share the resources from other departments when not in use.

== Access requests ==
This resource is free to use to users in the Departments of Statistics & Data Science, Philosophy, and CMIST. These departments set policies determining which of their users are eligible. Users outside these departments must be sponsored by someone in the eligible departments, as part of a project or collaboration.

To request access, complete the [https://docs.google.com/forms/d/e/1FAIpQLSftCsplE06whGocn62DhwC6NpDSgZ-DMmUacrQID_slIfg6sA/viewform access request form] and our team will be in touch to discuss your request and onboard you.

== Cluster setup ==
The cluster currently consists of 4 servers:
*The head node: You log in to this to write code and run jobs. It has 128 GB of RAM, 16 cores, and '''no GPU'''.
*The compute nodes: Jobs run on these. Each has 256 CPU cores, 8 NVIDIA L40 GPU cards with 48 GB of video RAM each, approximately 1 TB of RAM, and a total of approximately 3 TB of local data/scratch space.
*The file server: Holds 50TB of storage. The head node and compute nodes connect via fiberoptic networking. Your home directory is hosted on the file server, so the same files are accessible on the head node and all compute nodes.
The cluster is currently partitioned. One compute node is reserved for each department involved.

== Connecting and setting up your environment ==

=== SSH ===

From your terminal, type <code>ssh <YourStatID>@wright.hss.cmu.edu</code> . You can also avoid remembering this by adding the following snippet to your local <code>~/.ssh/config</code> file:
<pre>Host wright
Hostname wright.hss.cmu.edu
User YourStatID
</pre>

You can then connect by simply typing <code>ssh wright</code>.

This connects you to the '''head node.''' Here you can upload your data files, set up your scripts, and test things.

'''Do not run resource-intensive tasks directly on the head node'''. It doesn't have any GPUs, and it has fewer resources than the compute nodes. You can do quick tests to make sure your scripts are set up correctly, but submit all other jobs through Slurm.

=== Uploading files ===
To upload scripts or data files, you can use <code>scp</code> from your local computer (Mac or Linux); on Windows, you can get the Tectia SSH graphical client [https://www.cmu.edu/computing/services/endpoint/software/access.html free through CMU].

You can also connect through the [https://code.visualstudio.com/docs/remote/ssh Remote SSH feature of VS Code] if you like. This is useful to 

*navigate the file system from a proper GUI and 
*modify code/files/folder in place if you are not familiar with the command line
Files uploaded to your home directory will be accessible on the head node and all compute nodes, so they can be freely used in compute jobs.

=== Using Conda ===
The head node has Anaconda installed for managing Python packages and system libraries; see [[Python virtual environments]] for information on how to use it (but note it's '''already installed''', so you don't need to install it from scratch!).

However, the compute nodes '''do not''' have conda, so your slurm jobs can't use <code>conda activate</code> or other familiar commands. Instead, set up your environment on the head node, activate it, install the packages you need, and so on. Then run <code>which python</code> to get the path to the Python executable in your environment. For instance:<syntaxhighlight lang="bash">
$ which python
/home/export/areinhar/.conda/envs/llm/bin/python
</syntaxhighlight>This is the path you'll use in your scripts (see below).

=== Open OnDemand ===

Open OnDemand is a graphical interface where you can run jobs, interactive applications, and manage files. More information can be found [[Open OnDemand Wright|here]].

== Job scheduling and prioritization ==

The GPU cluster manages jobs via [[Main_Page/slurm|Slurm]]. Slurm is a job management system that deals with job prioritization and the allocation of resources. On high-performance computers (e.g., PSC and our GPU cluster) Slurm helps allocate resources so multiple jobs can be run quickly. Users run programs by submitting scripts to the Slurm job scheduler, and the scheduler runs them when resources are available.
=== Queue status ===

To check what jobs are running, use squeue:<syntaxhighlight lang="bash">
$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
1326 phil bash shaoanxi R 22:41:00 1 n02
1327 phil bash shaoanxi R 22:39:42 1 n02
1486 statds run_coun soheuny R 1:12:53 1 n01
1488 statds run_coun soheuny R 1:06:00 1 n01
1489 statds run_coun soheuny R 1:00:53 1 n01
1490 statds run_coun soheuny R 1:00:07 1 n01
1491 statds run_coun soheuny R 24:02 1 n01
1492 statds run_coun soheuny R 6:15 1 n01
1493 statds run_coun soheuny R 3:16 1 n01
1494 statds run_coun soheuny R 0:52 1 n01
</syntaxhighlight>

Here we see there are eight jobs running on node 1 and 2 jobs running on node 2.

To check what resources are currently in use, use <code>sinfo</code>:<syntaxhighlight lang="bash">
$ sinfo -O "NodeList:10,CPUsState:16,Memory:9,FreeMem:10,Gres:20,GresUsed:30"
NODELIST CPUS(A/I/O/T) MEMORY FREE_MEM GRES GRES_USED
n01 64/192/0/256 1000000 221943 gpu:nvidia-l40:8 gpu:nvidia-l40:8(IDX:0-7)
n02 32/224/0/256 1000000 225575 gpu:nvidia-l40:8 gpu:nvidia-l40:0(IDX:N/A)
</syntaxhighlight>This shows that node 1 (Statistics & Data Science) has 64 CPUs allocated and 192 idle, out of 256 total. There is 1TB of RAM, of which 222 GB is free. There are 8 NVIDIA L40 GPus, all 8 of which are in use. Hence if we submit a job to the statds partition requesting any GPUs, it will have to wait until some become free. Meanwhile, node 2 (Philosophy) has no GPUs in use.

=== Partitions ===
The cluster is divided into three partitions:

* <code>statds</code> (Statistics & Data Science)
* <code>phil</code> (Philosophy)
* <code>cmist</code> (CMIST)

Jobs are submitted to a partition. '''Users must submit jobs to their department's partition''' -- jobs submitted to the wrong partition will be rejected.

Each partition has its own physical server resources, so this ensures each department gets priority access to its own hardware.

=== Job prioritization ===
Slurm runs jobs based on the resources they request and what is available. If resources aren't available for your job, slurm will put it in the queue and wait until other jobs finish. However, it's not simply a first-in first-out queue -- Slurm attempts to arrange the jobs so every job can run, even if that means running them out of order. Slurm also adjusts each user's priority based on how many jobs they submit, so that one user can't hog the cluster by submitting dozens of jobs; if one user has many jobs waiting, theirs will be reduced in priority so other users can run jobs as well.

Examples:

* Someone submits Job A that requests 6 GPUs. Later, you submit Job B that requests 4 GPUs on the same partition. The compute node only has 8 GPUs, so Job B will be queued until Job A completes.
* Someone submits Job A that requests 6 GPUs. Later, you submit Job B that requests 2 GPUs. Job B can run immediately.
* You submit a job that requires all 8 GPUs on a node, but 4 are currently in use. Your job is put in the queue and scheduled to run when Slurm knows the current jobs will finish (because of their maximum time limit). Meanwhile, someone submits a job using 2 GPUs and a short time limit. Slurm lets it run immediately, because it knows it will be done before your job is scheduled to run. Other jobs may be put off until after yours runs, so yours can get the 8 GPUs it needs.
* You submit 50 jobs that each require 1 GPU. The node has only 8 GPUs, but several are in use. Several of your jobs run, but the others are queued for later. Other users can submit jobs while yours are in the queue, and some of their jobs may run before some of yours, for fairness.
You can view the currently running or queued jobs by running <code>squeue</code> on the head node, as described above. The <code>NODELIST(REASON)</code> column will indicate which jobs are queued by giving special codes:

* <code>(Resources)</code> means there are not enough resources (CPUs, memory, or GPUs) available for the job. It will run once the resources are free.
* <code>(Priority)</code> means resources are available, but Slurm wants to run a higher-priority job first. For example, maybe a large job has been waiting for resources for two days before you submit your job; Slurm may make your job wait so the large job can start first.

=== Submitting "condo" jobs ===
If your department's partition is busy, you may submit jobs to "condo" partitions to run on other nodes. The condo partitions are special:

* Your job will run at a lower priority than that department's users. (Slurm calls this "QOS", so the jobs will have "normal" as their priority, vs. a "priority" value for the department's users.)
* Your job will be preemptible: if the host department's users submit a job after yours that requires your resources, your job will be terminated and put back in the queue.

The condo partitions are named the same as the usual partitions, but ending with <code>_condo</code>. You must submit to the condo partition -- if you submit to another department's partition otherwise, your job will be rejected.

For example, if you are a Philosophy user, submitting a job to the <code>statds_condo</code> partition will run it on the Statistics node, but only when Statistics users are not using the resources. If a Statistics user submits a large job after yours, your job might be terminated, and will automatically run again when resources become available.

== Setting up jobs ==

=== Submitting batch jobs ===
The best way to run jobs is by writing a shell script specifying the Slurm options. To launch a batch job from a script you will need two files:

*the actual script in your favorite programming language that does whatever computation you want
*a bash script that requests the resources you need from Slurm and gives the commands to launch your job.
A typical bash script might look like this:<pre>#!/bin/bash
# specify partition to run in (statds, phil, or cmist)
#SBATCH --partition statds
#SBATCH --ntasks 1
#SBATCH --job-name=nameofyourjob

# number of CPUs you need:
#SBATCH --cpus-per-task 1

# amount of memory per CPU
# (total allocated is CPUs * memory)
#SBATCH --mem-per-cpu 4G

# number of GPUs you need
#SBATCH --gres=gpu:1

# maximum time limit for job, HH:MM:SS (up to 48 hours max, 48:00:00)
#SBATCH --time=01:00:00

# email you when job is finished or errors out:
#SBATCH --mail-user=YourAndrewID@andrew.cmu.edu
#SBATCH --mail-type=END,FAIL

## Choose one:

# to run a Python script in a Conda environment called ENV
srun /home/export/YOURANDREWID/.conda/envs/ENV/bin/python YourScript.py

# to run an R script:
srun Rscript YourScript.R
</pre>Adjust the number of CPUs, number of GPUs, memory usage, and time limit based on the needs of your script. Requesting more than you need may result in your job having to wait for resources to free up; requesting a long time you don't need will block slurm from scheduling other jobs for when yours will be done. (And if you request more resources than exist on the compute nodes, your job will be queued forever while it waits for resources to appear.)

'''Note:''' Slurm batch jobs are limited to a maximum duration of 48 hours. Jobs running longer will be killed.

To run the job, SSH into the head node and navigate to the directory with your scripts. Then run:<syntaxhighlight lang="bash">
sbatch example.sh
</syntaxhighlight>where <code>example.sh</code> is the name of your bash script.

You will see a message like "Submitted batch job 1209" when your job is submitted. Any printed output from your script will automatically be redirected to a file based on the job number -- in this case, <code>slurm-1209.out</code>. You can use <code>tail -f slurm-1209.out</code> to view the output live and monitor your script's progress.

=== Selecting the right resources ===
In the Slurm script above, you can choose the number of tasks, the number of GPUs, the memory per CPU, the number of GPUs, and the time limit. Choose these values wisely:

* You generally want '''1 task''' (<code>--ntasks 1</code>). Asking for multiple tasks will cause srun to run multiple copies of your script simultaneously. If you need to run the same script with 10 different parameter values, this can be useful: request 10 tasks, and read the <code>SLURM_PROCID</code> environment variable in your script. It will be numbered 0-9 (for 10 tasks), so you can use this to set the parameter values for each parallel run.
* Set the number of CPUs based on your script. If your script is entirely single-threaded, ask for 1. If you use parallel processing (such as with multiprocessing in Python or the parallel package in R), use a multithreaded package, or spawn many parallel programs in your job, request more.
* Request the minimum amount of memory (RAM) needed for your script. This is per-CPU, so requesting a lot of CPUs and RAM can easily mean you request more RAM than exists on the node.
* Start with 1 GPU, adding more if your job requires more GPU memory (VRAM) than is on one GPU, or if you explicitly write your script to use more. If you do not use GPUs, request 0 so you aren't allocated a GPU that nobody else can use while your job runs.
* Set the time limit a little longer than you expect your script to run. Slurm uses the time limit when scheduling, so always choosing the maximum (48 hours) will backfire: Slurm will assume your job will take a long time and will run other things first. If you know your job can only take an hour, setting, say, a 01:30:00 time limit will let Slurm fit your job in before other long jobs in the queue. But of course if you choose too low and your job exceeds the limit, it will be killed, so leave yourself some extra time.

=== Interactive mode ===

You can also run interactive sessions that allow you to, say, run an R or Python console and type commands and immediately view the output. From the head node, run a command like:<syntaxhighlight lang="bash">
# remember to edit to correct partition:
srun --partition statds --ntasks 1 \
--cpus-per-task 16 --mem-per-cpu 4G \
--gres=gpu:1 --time=01:00:00 --pty \
bash
</syntaxhighlight>This runs a job that opens a bash shell on the compute node, and connects you directly to this job. You can now run bash commands, open Python, or run R scripts. Note your interactive session has a time limit and will be killed when it elapses, so set the time limit carefully and choose the resource amounts in the command according to your needs.

Interactive sessions are limited to '''two hours maximum'''. You cannot request a longer time limit. This is because interactive sessions are inefficient, with the cluster spending most of its time waiting for you to type something instead of running code. For large tasks, submit a batch job.

=== Interactive via VS Code ===
VS Code supports SSH and tunneling, which allows it to connect to a server and run commands. This allows you to use the Jupyter Notebook support built into VS Code to interact with the GPU cluster. As this uses an interactive session, it's subject to the same '''two-hour time limit'''. Hence notebooks should be used only for exploration and debugging, not large jobs.

To use this, we need to connect to the head node, run an interactive Slurm session, and then connect VS Code to that Slurm session so it can use it. This takes a few extra steps.

==== Setup ====
Before you use this for the first time, do the following steps:

*SSH into Wright and run the following to install and unpack the VSCode CLI: 
<pre>curl -Lk 'https://code.visualstudio.com/sha/download?build=stable&os=cli-alpine-x64' --output vscode_cli.tar.gz
tar -xf vscode_cli.tar.gz
</pre>

*Add the <code>code</code> executable to your PATH by adding the following to your <code>.bashrc</code> file: <code>export PATH="~:$PATH"</code><pre>echo 'export PATH="$PATH:/path/to/code"' >> ~/.bashrc
</pre>
*In VSCode, install the <code>Remote - Tunnels</code> extension, then use the VSCode SSH feature to connect to Wright and make sure to install all the local VS Code extensions on the remote machine as well (e.g., Python, Jupyter, ...)

==== Running a session ====
*In the Wright headnode, launch a Slurm interactive session with <code>srun</code>, following the instructions above.

*Inside the interactive session, run <code>code tunnel</code> and follow the instructions. The first time the system will ask you to name the tunnel and authorize the process via an authenticator, I suggest using GitHub since it usually works quite smoothly. Subsequent tunnels might still ask you to authorize their creation by typing an alphanumerical sequence into a link that appears in the terminal. Once you’ve done that, the tunnel is successfully created.
*From VSCode, select <code>Connect to Tunnel</code>, then choose the authentication method you previously used and you will see the tunnel appear in the dropdown. Select it and you will land directly into the interactive session we just created
*You will now be able to run code and, e.g., Jupyter Notebooks using the compute node allocated by the Slurm interactive session. VS Code will be disconnected when the Slurm session ends, such as when it reaches its time limit.
Note that if you need to use a conda environment, you'll have to configure VS Code to use the Python executable you found above, since conda is not installed on the compute nodes.

=== SSH to Compute Nodes ===
The ability to SSH directly to compute nodes is disabled. You can run the command below to start a bash terminal on a compute node when running a batch job. This allows for the same functionality as SSH.<syntaxhighlight lang="bash">
srun —jobid=yourjobidhere --pty /bin/bash
</syntaxhighlight>

Wright GPU Cluster

2026-01-20T16:49:52Z

Areinhar: improve organization

The department has access to a powerful GPU-based cluster named [https://en.wikipedia.org/wiki/Sewall_Wright Wright] which is hosted here at CMU. This is a 4-node and growing cluster that is a shared resource for Statistics & Data Science, Philosophy, and other departments in Dietrich College. Each department has priority access to its own compute node, but will be able to share the resources from other departments when not in use.

== Access requests ==
This resource is free to use to users in the Departments of Statistics & Data Science, Philosophy, and CMIST. These departments set policies determining which of their users are eligible. Users outside these departments must be sponsored by someone in the eligible departments, as part of a project or collaboration.

To request access, complete the [https://docs.google.com/forms/d/e/1FAIpQLSftCsplE06whGocn62DhwC6NpDSgZ-DMmUacrQID_slIfg6sA/viewform access request form] and our team will be in touch to discuss your request and onboard you.

== Cluster setup ==
The cluster currently consists of 4 servers:
*The head node: You log in to this to write code and run jobs. It has 128 GB of RAM, 16 cores, and '''no GPU'''.
*The compute nodes: Jobs run on these. Each has 256 CPU cores, 8 NVIDIA L40 GPU cards with 48 GB of video RAM each, approximately 1 TB of RAM, and a total of approximately 3 TB of local data/scratch space.
*The file server: Holds 50TB of storage. The head node and compute nodes connect via fiberoptic networking. Your home directory is hosted on the file server, so the same files are accessible on the head node and all compute nodes.
The cluster is currently partitioned. One compute node is reserved for each department involved.

== Connecting and setting up your environment ==

=== SSH ===

From your terminal, type <code>ssh <YourStatID>@wright.hss.cmu.edu</code> . You can also avoid remembering this by adding the following snippet to your local <code>~/.ssh/config</code> file:
<pre>Host wright
Hostname wright.hss.cmu.edu
User YourStatID
</pre>

You can then connect by simply typing <code>ssh wright</code>.

This connects you to the '''head node.''' Here you can upload your data files, set up your scripts, and test things.

'''Do not run resource-intensive tasks directly on the head node'''. It doesn't have any GPUs, and it has fewer resources than the compute nodes. You can do quick tests to make sure your scripts are set up correctly, but submit all other jobs through Slurm.

=== Uploading files ===
To upload scripts or data files, you can use <code>scp</code> from your local computer (Mac or Linux); on Windows, you can get the Tectia SSH graphical client [https://www.cmu.edu/computing/services/endpoint/software/access.html free through CMU].

You can also connect through the [https://code.visualstudio.com/docs/remote/ssh Remote SSH feature of VS Code] if you like. This is useful to 

*navigate the file system from a proper GUI and 
*modify code/files/folder in place if you are not familiar with the command line
Files uploaded to your home directory will be accessible on the head node and all compute nodes, so they can be freely used in compute jobs.

=== Using Conda ===
The head node has Anaconda installed for managing Python packages and system libraries; see [[Python virtual environments]] for information on how to use it (but note it's '''already installed''', so you don't need to install it from scratch!).

However, the compute nodes '''do not''' have conda, so your slurm jobs can't use <code>conda activate</code> or other familiar commands. Instead, set up your environment on the head node, activate it, install the packages you need, and so on. Then run <code>which python</code> to get the path to the Python executable in your environment. For instance:<syntaxhighlight lang="bash">
$ which python
/home/export/areinhar/.conda/envs/llm/bin/python
</syntaxhighlight>This is the path you'll use in your scripts (see below).

=== Open OnDemand ===

Open OnDemand is a graphical interface where you can run jobs, interactive applications, and manage files. More information can be found [[Open OnDemand Wright|here]].

== Job scheduling and prioritization ==

The GPU cluster manages jobs via [[Main_Page/slurm|Slurm]]. Slurm is a job management system that deals with job prioritization and the allocation of resources. On high-performance computers (e.g., PSC and our GPU cluster) Slurm helps allocate resources so multiple jobs can be run quickly. Users run programs by submitting scripts to the Slurm job scheduler, and the scheduler runs them when resources are available.
=== Queue status ===

To check what jobs are running, use squeue:<syntaxhighlight lang="bash">
$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
1326 phil bash shaoanxi R 22:41:00 1 n02
1327 phil bash shaoanxi R 22:39:42 1 n02
1486 statds run_coun soheuny R 1:12:53 1 n01
1488 statds run_coun soheuny R 1:06:00 1 n01
1489 statds run_coun soheuny R 1:00:53 1 n01
1490 statds run_coun soheuny R 1:00:07 1 n01
1491 statds run_coun soheuny R 24:02 1 n01
1492 statds run_coun soheuny R 6:15 1 n01
1493 statds run_coun soheuny R 3:16 1 n01
1494 statds run_coun soheuny R 0:52 1 n01
</syntaxhighlight>

Here we see there are eight jobs running on node 1 and 2 jobs running on node 2.

To check what resources are currently in use, use <code>sinfo</code>:<syntaxhighlight lang="bash">
$ sinfo -O "NodeList:10,CPUsState:16,Memory:9,FreeMem:10,Gres:20,GresUsed:30"
NODELIST CPUS(A/I/O/T) MEMORY FREE_MEM GRES GRES_USED
n01 64/192/0/256 1000000 221943 gpu:nvidia-l40:8 gpu:nvidia-l40:8(IDX:0-7)
n02 32/224/0/256 1000000 225575 gpu:nvidia-l40:8 gpu:nvidia-l40:0(IDX:N/A)
</syntaxhighlight>This shows that node 1 (Statistics & Data Science) has 64 CPUs allocated and 192 idle, out of 256 total. There is 1TB of RAM, of which 222 GB is free. There are 8 NVIDIA L40 GPus, all 8 of which are in use. Hence if we submit a job to the statds partition requesting any GPUs, it will have to wait until some become free. Meanwhile, node 2 (Philosophy) has no GPUs in use.

=== Partitions ===
The cluster is divided into three partitions:

* <code>statds</code> (Statistics & Data Science)
* <code>phil</code> (Philosophy)
* <code>cmist</code> (CMIST)

Jobs are submitted to a partition. '''Users must submit jobs to their department's partition''' -- jobs submitted to the wrong partition will be rejected.

Each partition has its own physical server resources, so this ensures each department gets priority access to its own hardware.

=== Job prioritization ===
Slurm runs jobs based on the resources they request and what is available. If resources aren't available for your job, slurm will put it in the queue and wait until other jobs finish. However, it's not simply a first-in first-out queue -- Slurm attempts to arrange the jobs so every job can run, even if that means running them out of order. Slurm also adjusts each user's priority based on how many jobs they submit, so that one user can't hog the cluster by submitting dozens of jobs; if one user has many jobs waiting, theirs will be reduced in priority so other users can run jobs as well.

Examples:

* Someone submits Job A that requests 6 GPUs. Later, you submit Job B that requests 4 GPUs on the same partition. The compute node only has 8 GPUs, so Job B will be queued until Job A completes.
* Someone submits Job A that requests 6 GPUs. Later, you submit Job B that requests 2 GPUs. Job B can run immediately.
* You submit a job that requires all 8 GPUs on a node, but 4 are currently in use. Your job is put in the queue and scheduled to run when Slurm knows the current jobs will finish (because of their maximum time limit). Meanwhile, someone submits a job using 2 GPUs and a short time limit. Slurm lets it run immediately, because it knows it will be done before your job is scheduled to run. Other jobs may be put off until after yours runs, so yours can get the 8 GPUs it needs.
* You submit 50 jobs that each require 1 GPU. The node has only 8 GPUs, but several are in use. Several of your jobs run, but the others are queued for later. Other users can submit jobs while yours are in the queue, and some of their jobs may run before some of yours, for fairness.
You can view the currently running or queued jobs by running <code>squeue</code> on the head node, as described above. The <code>NODELIST(REASON)</code> column will indicate which jobs are queued by giving special codes:

* <code>(Resources)</code> means there are not enough resources (CPUs, memory, or GPUs) available for the job. It will run once the resources are free.
* <code>(Priority)</code> means resources are available, but Slurm wants to run a higher-priority job first. For example, maybe a large job has been waiting for resources for two days before you submit your job; Slurm may make your job wait so the large job can start first.

=== Submitting "condo" jobs ===
If your department's partition is busy, you may submit jobs to "condo" partitions to run on other nodes. The condo partitions are special:

* Your job will run at a lower priority than that department's users. (Slurm calls this "QOS", so the jobs will have "normal" as their priority, vs. a "priority" value for the department's users.)
* Your job will be preemptible: if the host department's users submit a job after yours that requires your resources, your job will be terminated and put back in the queue.

The condo partitions are named the same as the usual partitions, but ending with <code>_condo</code>. You must submit to the condo partition -- if you submit to another department's partition otherwise, your job will be rejected.

For example, if you are a Philosophy user, submitting a job to the <code>statds_condo</code> partition will run it on the Statistics node, but only when Statistics users are not using the resources. If a Statistics user submits a large job after yours, your job might be terminated, and will automatically run again when resources become available.

== Setting up jobs ==

=== Submitting batch jobs ===
The best way to run jobs is by writing a shell script specifying the Slurm options. To launch a batch job from a script you will need two files:

*the actual script that implements things in your favorite programming language
*a bash script that requests the resources you need from Slurm and gives the commands to launch your job.
A typical bash script might look like this:<pre>#!/bin/bash
# specify partition to run in (statds, phil, or cmist)
#SBATCH --partition statds
#SBATCH --ntasks 1
#SBATCH --job-name=nameofyourjob

# number of CPUs you need:
#SBATCH --cpus-per-task 1

# amount of memory per CPU
# (total allocated is CPUs * memory)
#SBATCH --mem-per-cpu 4G

# number of GPUs you need
#SBATCH --gres=gpu:1

# maximum time limit for task (up to 48 hours max, 48:00:00)
#SBATCH --time=01:00:00

# email you when job is finished or errors out:
#SBATCH --mail-user=YourAndrewID@andrew.cmu.edu
#SBATCH --mail-type=END,FAIL

## Choose one:

# to run a Python script in a Conda environment called ENV
srun /home/export/YOURANDREWID/.conda/envs/ENV/bin/python YourScript.py

# to run an R script:
srun Rscript YourScript.R
</pre>Adjust the number of CPUs, number of GPUs, memory usage, and time limit based on the needs of your script. Requesting more than you need may result in your job having to wait for resources to free up; requesting a long time you don't need will block slurm from scheduling other jobs for when yours will be done. (And if you request more resources than exist on the compute nodes, your job will be queued forever while it waits for resources to appear.)

'''Note:''' Slurm batch jobs are limited to a maximum duration of 48 hours. Jobs running longer will be killed.

To run the job, SSH into the head node and navigate to the directory with your scripts. Then run:<syntaxhighlight lang="bash">
sbatch example.sh
</syntaxhighlight>where <code>example.sh</code> is the name of your bash script.

You will see a message like "Submitted batch job 1209" when your job is submitted. Any printed output from your script will automatically be redirected to a file based on the job number -- in this case, <code>slurm-1209.out</code>. You can use <code>tail -f slurm-1209.out</code> to view the output live and monitor your script's progress.

=== Interactive mode ===

You can also run interactive sessions that allow you to, say, run an R or Python console and type commands and immediately view the output. From the head node, run a command like:<syntaxhighlight lang="bash">
# remember to edit to correct partition:
srun --partition statds --ntasks 1 \
--cpus-per-task 16 --mem-per-cpu 4G \
--gres=gpu:1 --time=01:00:00 --pty \
bash
</syntaxhighlight>This runs a job that opens a bash shell on the compute node, and connects you directly to this job. You can now run bash commands, open Python, or run R scripts. Note your interactive session has a time limit and will be killed when it elapses, so set the time limit carefully and choose the resource amounts in the command according to your needs.

Interactive sessions are limited to '''two hours maximum'''. You cannot request a longer time limit. This is because interactive sessions are inefficient, with the cluster spending most of its time waiting for you to type something instead of running code. For large tasks, submit a batch job.

=== Interactive via VS Code ===
VS Code supports SSH and tunneling, which allows it to connect to a server and run commands. This allows you to use the Jupyter Notebook support built into VS Code to interact with the GPU cluster. As this uses an interactive session, it's subject to the same '''two-hour time limit'''. Hence notebooks should be used only for exploration and debugging, not large jobs.

To use this, we need to connect to the head node, run an interactive Slurm session, and then connect VS Code to that Slurm session so it can use it. This takes a few extra steps.

==== Setup ====
Before you use this for the first time, do the following steps:

*SSH into Wright and run the following to install and unpack the VSCode CLI: 
<pre>curl -Lk 'https://code.visualstudio.com/sha/download?build=stable&os=cli-alpine-x64' --output vscode_cli.tar.gz
tar -xf vscode_cli.tar.gz
</pre>

*Add the <code>code</code> executable to your PATH by adding the following to your <code>.bashrc</code> file: <code>export PATH="~:$PATH"</code><pre>echo 'export PATH="$PATH:/path/to/code"' >> ~/.bashrc
</pre>
*In VSCode, install the <code>Remote - Tunnels</code> extension, then use the VSCode SSH feature to connect to Wright and make sure to install all the local VS Code extensions on the remote machine as well (e.g., Python, Jupyter, ...)

==== Running a session ====
*In the Wright headnode, launch a Slurm interactive session with <code>srun</code>, following the instructions above.

*Inside the interactive session, run <code>code tunnel</code> and follow the instructions. The first time the system will ask you to name the tunnel and authorize the process via an authenticator, I suggest using GitHub since it usually works quite smoothly. Subsequent tunnels might still ask you to authorize their creation by typing an alphanumerical sequence into a link that appears in the terminal. Once you’ve done that, the tunnel is successfully created.
*From VSCode, select <code>Connect to Tunnel</code>, then choose the authentication method you previously used and you will see the tunnel appear in the dropdown. Select it and you will land directly into the interactive session we just created
*You will now be able to run code and, e.g., Jupyter Notebooks using the compute node allocated by the Slurm interactive session. VS Code will be disconnected when the Slurm session ends, such as when it reaches its time limit.
Note that if you need to use a conda environment, you'll have to configure VS Code to use the Python executable you found above, since conda is not installed on the compute nodes.

=== SSH to Compute Nodes ===
The ability to SSH directly to compute nodes is disabled. You can run the command below to start a bash terminal on a compute node when running a batch job. This allows for the same functionality as SSH.<syntaxhighlight lang="bash">
srun —jobid=yourjobidhere --pty /bin/bash
</syntaxhighlight>

Terminal multiplexers

2025-10-24T15:13:58Z

Areinhar: Formatting improvements

“Terminal Multiplexers”, like <code>screen</code> and <code>tmux</code>, are extremely useful tools for running jobs on the department servers. The primary reason to use them is so you can run a job on the server and not need to keep your connection with the server open (e.g. you can close your computer, go to sleep, etc. while your job runs). It’s often the case that your connection to the server might disconnect even if your computer just goes to sleep or disconnects from the internet for a minute.

There are two major terminal multiplexer options: <code>screen</code> and <code>tmux</code>. The Internet recommends <code>tmux</code> (but some people in the department use <code>screen</code>).

In terms of usage, the most important commands 1. creating a new session 2. detaching but not closing as session 3. reattaching a session and 4. closing a session.

With these four options you can start up a session (1), step away from a session / the server without killing a job you’re running (2), come back to the session you left (3) and close a session once you’re done with it (4). The rest of the commands are icing on the cake.

In the following sections below, we’ll present pros of both, then present sections on standard commands for both approaches. This article is currently more a place to remind yourself of what commands to use than a new introduction to the ideas. We recommend reading either of the introductions (at the end of either tool’s subsection) for get exposure to the power of these tools.

== Comparisons ==

#<code>tmux</code>:
#*more consistent syntax (but <code>screen</code> is still pretty good)
#*preserves panels / windows even when you detach the session
#*maybe a little safer in closing sessions (confirms request)

#<code>screen</code>:
#*a stable tool (<code>tmux</code> is a little more in flux)
== <code>screen</code> ==

=== Basic commands ===

{| class="wikitable"
|-
! width="37%" | Action
! width="33%" | Command
! width="29%" | Notes
|-
| '''Start new session [important]'''
| <code>screen</code>
|  
|-
| Starting a named session
| <code>screen -S _session_name_</code>
|  
|-
| List current sessions
| <code>screen -list</code>
|  
|-
| '''Detach session [important]'''
| <code>(Ctrl+a)+d</code>
|  
|-
| '''Attach/ reopen session [important]'''
| <code>screen -r _session_name_</code>
| If only one session is open you can just use <code>screen -r</code>
|-
| '''Close session [important]'''
| <code>Ctrl+d</code>
|  
|-
| '''Kill a session you are not currently in'''
| <code>screen -XS _session_name_ quit</code>
|  
|-
| enter scrollback/copy mode
| <code>Ctrl+a+[</code>
|  
|-
| end scrollback/paste buff
| <code>Ctrl+a+]</code>
|  
|}

=== Window/panel based commands ===

Having multiple panels / windows going in a single session at the same time.

{| class="wikitable"
|-
! width="37%" | Action
! width="33%" | Command
! width="29%" | Notes
|-
| Create new window
| <code>(Ctrl+a)+c</code>
|  
|-
| split terminal horizontally, and create a new window
| <code>(Ctrl+a)+S</code>  +  <code>(Ctrl+a)+c</code>
| Note the capital S. 
|-
| split terminal vertically, and create a new window
| <code>(Ctrl+a)+| </code>+  <code>(Ctrl+a)+c</code>
| This only works if the screen is a high enough version.
|-
| move between windows
| <code>(Ctrl+a)+</code> …
|  
|-
| - next window
| <code>+n</code>
|  
|-
| - previous window
| <code>+p</code>
|  
|-
| - pick from list
| <code>+"</code>
|  
|-
| Move between splits
| <font face="monospace">(Ctrl+a)+ Tab</font>
|  
|}

=== External Links ===

*Introduction: [https://www.howtogeek.com/662422/how-to-use-linuxs-screen-command/ howtogeek.com], [https://linuxize.com/post/how-to-use-linux-screen/ linuxize.com]
*Cheat sheets: [http://www.pixelbeat.org/lkdb/screen.html pixelbeat.org].

== <code>tmux</code> ==

=== Basic commands ===

{| class="wikitable"
|-
! width="37%" | Action
! width="33%" | Command
! width="29%" | Notes
|-
| '''Start session [important]'''
| <code>tmux</code>
|  
|-
| Start named session
| <code>tmux new -s _session_name_</code>
|  
|-
| Rename session
| <code>(Ctrl+b)+:</code> -> then <code>rename-session -t _current_name_ _new_name_</code>
| You don’t have to include <code>-t _current_name_</code> if you want to rename a session you are in.
|-
| '''Detach session [important]'''
| <code>(Ctrl+b)+d</code>
|  
|-
| '''Attach/ reopen session [important]'''
| <code>tmux attach-session -t _session_name_</code>
|  
|-
| '''Close session [important]'''
| <code>tmux kill-session -t _session_name_</code>
|  
|-
| List current sessions
| <code>tmux ls</code>
|  
|}

=== Window/panel based commands ===

Having multiple panels / windows going in a single session at the same time.

{| class="wikitable"
|-
! Action
! Command
! Notes
|-
| new window
| <code>(Ctrl+b)+c</code>
|  
|-
| see window numbering
| <code>(Ctrl+b)+q</code>
| flashes number of each panel open
|-
| close current window
| <code>(Ctrl+b)+&</code> -> then <code>y</code> close current window
|  
|-
| create panel splitting
| <code>(Ctrl+b)+</code> …
|  
|-
| - horizontal splitting
| <code>+”</code>
| this is capital<code>"</code>
|-
| - vertical splitting
| <code>+%</code>
| this is also a capital <code>%</code>
|-
| - move between panels
| <code>+</code> arrow keys
|  
|-
| move between windows
| <code>(Ctrl+b)+</code> …
|  
|-
| - next window
| <code>+n</code>
|  
|-
| - previous window
| <code>+p</code>
|  
|  
|}

=== External Links ===

*Introduction: [https://www.howtogeek.com/671422/how-to-use-tmux-on-linux-and-why-its-better-than-screen/ howtogeek.org]
*Cheat sheets: [https://www.howtogeek.com/671422/how-to-use-tmux-on-linux-and-why-its-better-than-screen/ howtogeek.org] (see the bottom of the article).

Main Page

2025-10-24T15:13:08Z

Areinhar: /* Servers */ Mention Leviathans

Anyone can read this wiki.  If you want to write/edit stuff in the wiki, you will need to log in with your account provided by the Statistics & Data Science Department computing group. This Wiki is intended to be a repository for instruction, information and ideas related to the many computing environments available to members and affiliates of the Statistics & Data Science Department.

== Getting to know the computing resources & getting support ==

If you're interested in knowing about what resources you have access to (and/or) what resources you have access to that might solve a problem you're having, please visit our [[What_resources_when|"what resources when" article]].

To get support that is beyond this website please attend office hours below and/or email [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu] for support (for more details about how the support ticketing works through [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu] + ServiceNow please see the [[Service_now|ServiceNow]] article).

Reset your StatDS password here - https://util.stat.cmu.edu/passwordreset/

== Department Contacts ==

• [mailto:events@stat.cmu.edu events@stat.cmu.edu]: seminars, conferences & workshops, and calendar updates

• [mailto:news@stat.cmu.edu news@stat.cmu.edu]: dept/dept member-related public news & announcements

• [mailto:lists@stat.cmu.edu lists@stat.cmu.edu]: department email lists

• [mailto:webmaster@stat.cmu.edu webmaster@stat.cmu.edu]: website updates

• [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu]: computer/tech requests

• [mailto:rooms@stat.cmu.edu rooms@stat.cmu.edu]: request space for meetings and TA office hours

• [mailto:reimbursements@stat.cmu.edu reimbursements@stat.cmu.edu]: travel and reimbursement requests

== Computing Office Hours ==

These office hour opportunities provide a dedicated time for you to come ask questions about research computing.  These include code advice, how to access certain resources, what the best resources are for your project, or to learn about new connections to help you progress in your education and research. Staff Office hour times will be sent out each week on Monday as a calendar invite; accept any of the invitations to add it to your personal calendar. PhD TA Office Hours are below, unless a different time is communicated.

#PhD TA Office Hours for the first half of summer 2025 are held by Elizabeth Cucuzzella in FMS Room 320 on Mondays from 2:30 pm to 4:30 pm. PhD students can also message on the #computing channel on slack.
#Staff Office Hours are held by appointment by Carl Skipper in BH 129G. Schedule via [mailto:lists@stat.cmu.edu email].

See [[Computing_office_hour_usage|computing office hour usage]] for what office hours can be used for.

== Computing Wiki Pages ==

=== Managing servers and environments ===

==== Introductory Tutorials ====
[[Basic Linux Commands|Introduction to Basic Linux Commands]]

==== Virtual environments ====
*[[Virtual Environment Introduction|Why use virtual environments?]]
*[https://info.stat.cmu.edu/index.php?title=Mamba Mamba (Conda) environments].
*R's [[Renv]]
*Python's [[Python_virtual_environments|virtual environments (2 options)]]
*Old instructions for [[Conda with Python (Reference Only)|conda with Python]]

==== Containers ====
*[[Introduction to containers]]
*[[Docker, and why-you-should-use-it-for-your-research]]
*[[Tutorials|Basic tutorial]]
*[[Running Docker locally and on Hydras]]

=== Department computing resources ===

==== Printing ====
*[[Department_Printers|Department Printers]]

==== Web Hosting ====
#[[How to host your webpage under stat.cmu.edu]]
#[[Bulding your website with Hugo]]

==== Servers ====
*[[Department Servers]].
*#[[Linux and department servers|Using the Servers Overview]] -This wiki introduces you to the Leviathan and Hydra servers, basic Linux commands, and more.
*#[[Terminal_multiplexers|Terminal multiplexers (screen and tmux)]]. This wiki presents tools for use on the servers that can allow you to run jobs even when your not on the server, and much much more!
*#[[Python on the server]]. This wiki introduces uses command line approaches to running <code>python</code> on a virtual machine, and how to use jupyter notebooks through the server as well.
*#[[R on the server]]. This wiki introduces uses command line approaches to running <code>R</code> on a virtual machine
*#[https://info.stat.cmu.edu/index.php?title=Department_Servers#Software Software] - overview of the software installed and ready for use.
*#[https://info.stat.cmu.edu/index.php?title=Monitoring_with_Grafana Monitor Server Usage]

*[[Department GPU Servers]], for small jobs needing 1 GPU.
*[[GPU Cluster]], for larger jobs that may need multiple GPUs and many CPUs.
*[[PSC]], the Pittsburgh Supercomputing Center. Free accounts and usage credits are available
*#[[ACCESS]]
*#[[Slurm]]. This wiki introduces the job management system uses at PSC.
*#[[Parallelization_on_PSC/Bridges-2]]
*#[[Access PSC compute nodes directly from VSCode]]
*[[Open OnDemand]]

=== Development guides ===

====[[Parallel_computing|Parallelization]]====
*[[Parallelization_in_r|R-specific parallelization]]
*[[Parallelization_in_python|Python-specific parallelization]]

==== Package development ====
*[[Developing_R_packages|R package development]]
*[[Developing_python_packages|Python package development]]
[[Terminal automation]]. This wiki introduces approaches to automate Linux/OS command line calls.
*[[Bash files]]. This wiki presents potential uses of Bash files including automated file/folder creation, parallelization and much more!
*[[Make]]. This wiki presents potential uses of <code>make</code> and Makefiles including automated workflow and dependency tracking
*[[Jumping to frequently used folders and automating environment setups]]

==== Version Control ====
See the guide provided in the site of [https://36-750.github.io/tools/version-control/ 36750 - Statistical Computing]

==== Reference management ====
#Zotero
#Mendeley

==== Integrated Development Environments (IDEs) ====
#[[VSCode|VSCode]]
#PyCharm
#RStudio
#Eclipse

==== Text Editors ====
#Vim
#Nano
#Emacs
#Atom
#Sublime

See also the guide provided in the site of [https://36-750.github.io/tools/editors/ 36750 - Statistical Computing]

==== AI Tools ====
* [[ChatGPT and other LLMs]]

*[[github_copilot|GitHub Copilot]]

==== Miscellaneous====
#[[Windows_Subsystem_for_Linux_(WSL)_and_Python|Windows Subsystem for Linux (WSL) and Python]]
#[[Amphetamine|Amphetamine: How to keep your computer awake]]

===General Department Computing Resources===

*Fall 2021 [[Computing_seminar_resources|Computing Seminar Resources]] (video and slides)

==Ideas for the Wiki==

Please see the [[TODO|TODO page]] to post suggestions and errata.

==Using the Wiki==

*Here is some [[Generic_Wiki_Information|Generic Wiki Information]] that you may be interested in, such as a users' guide and other resources for Wiki users.

== Archived Pages ==
These area pages that are potentially useful for some users, but no longer reflect the computing resources of the department writ large.

Azure
#[[Azure_getting_started|Azure: getting started]]. This wiki walks you through the steps to create a virtual machine through Azure
#[[How to obtain a virtual machine with a GPU]]
#[[Attach a disk to a virtual machine]]

What resources when

2025-10-24T15:12:25Z

Areinhar:

To try to answer the common question “what resources do I have access to and when should I use them”, below we present to approaches to answering the question. The first of which focuses on what resources exist and what they should be used for and the second tries to answer resource needs / current computing problems with resources to fit those needs/ solve those problems.

After you’re found the resource you think you’d like to use (or even when you’re still deciding), we strongly encourage you to attend an computing office hour to help discuss all the things you should be thinking about and get you on the right track quickly.

== What computing resources do you have access to ==

=== Your personal computer ===

Your personal computer is great for small jobs that take less than 30 minutes, uses few cores and not too much RAM.

'''Pro:'''

#Naturally the easiest tool to use since it’s yours. Best for short jobs that require only a few cores.

'''Cons:'''

#
capacity:
*few cores: 4-8 cores common on laptops<ref><p>From [http://laptopmag.com/articles/Cpu-comparison laptopmag]</p></ref> (hyperthreading means 8-16 “cores”).
*low RAM: 4-32 GB of common on laptops<ref><p>From [http://digitaltrends.com/computing/how-much-ram digitaltrends]</p></ref>, used across all cores
*“no” GPUs: you don’t have GPUs that are optimized for Deep Learning

#
other usage: You often want to use your computer for other things (and it’s built for that exact thing). It’s possible you or someone you know has has had to leave their laptop at home for the day/ run code overnight. When they need to run something intensive -OR- long. But you need that laptop for other work / homework…

The below XKCD comic doesn’t have to be you… (though breaks are good):

[[File:Compiling.png|thumb|none|https://xkcd.com/303/]]

'''Conclusion:''' Your personal computer isn’t the best for when doing a lot of computing and its easy to put jobs elsewhere!

'''Common use-cases'''

#Most class projects
#Non-data problems
#Non-complex structure problems

'''Associated Articles''':

*[[Parallel_computing|parallelization]]

=== The department servers ===

The departmental servers should be seen as a “sandbox”. You can experiment with lots of things on the servers and it doesn’t cost you much to do. The Leviathans and Hydras have more RAM than a typical laptop. The Leviathans also have top-of-the-line CPUs that should be faster than most laptops, though the Hydras have older CPUs that may not be faster than running code on your own laptop.

'''Pros:'''

#Best for long
#Plenty of RAM per machine (128 GB on Leviathans + hydra7 and 8, 64 GB on other hydras)
#Very low barrier to entry (aka easy to get started)
#File management system shares across all servers

'''Cons:'''

#usage: The department server is a bit of a “wild-west” in terms of usage (it varies how many people use it at a given time)
#cores per server: the number of cores per server is only 12 (Hydras) or 16 (Leviathans)

'''Conclusion:''' Great place to start for more intensive computing.

'''Associated Articles''':

*[[Linux_and_department_servers|Linux and department servers]]
'''Server Monitoring'''

* [https://info.stat.cmu.edu/grafana/d/beuw1q6y3dgjkc/hydra-cluster?orgId=1 Hydra Cluster Usage Monitor]
* [https://info.stat.cmu.edu/grafana/d/asjkldfbnvajdsfh213/leviathan-cluster?orgId=1 Leviathan Cluster Usage Monitor]

== PSC (Pittsburgh supercomputer center) ==

The Pittsburgh supercomputer center provides supercomputers than have massive amounts of cores (but also can find some with large RAMs and GPUs). Generally speaking, the supercomputing resources are often used when you have extremely parallelizable code that would take a very long time if you ran it on just a few cores (e.g. your computer or a few hydras).

'''Pros:'''

#A massive amount of cores and RAM that can be leveraged for extremely parallelizable computing
#Easier to access in the statistics department

'''Cons:'''

#Requires learning of new job management system (though quick to learn - see [Slurm])
#Some options to access the resources can take time or require going through your advisor
#''currently (2021)'' has been having problems with servers crashing

'''Associated Articles''':

*[[PSC|PSC]]
*[[Slurm|Slurm]]

== GPU Computing ==
The Statistics & Data Science Department offers two main GPU computing environments.

'''Ghidorahs'''

* 3 physical servers
* RTX 4090 GPUs
* Connected to the Stats network share
* Your statistics directory account will grant you access

'''Wright'''

* For a detail overview, please check [[GPU Cluster|here]]
* You must request access
* The Statistics network share is not accessible

== Computing resource flowchart ==

These flow charts try to help capture what’s available and when you might use them.

Visit the links below to view current computing resource usage. Select a server with the least amount of users and lowest resource usage.

* [https://info.stat.cmu.edu/grafana/d/fdkb6l4klj400d/hydra-cluster?orgId=1 Hydra Cluster Usage Monitor]

* [https://info.stat.cmu.edu/grafana/d/edkar1rgy3l6od/ghidorah-cluster?orgId=1 Ghidorah Cluster Usage Monitor]
[[File:ProblemsOnYourPersonalComputerUpdated.png|center|thumb|1000x1000px]]

From [http://laptopmag.com/articles/Cpu-comparison laptopmag]
<br/>      2. <font color="blue">↑</font>
From [http://digitaltrends.com/computing/how-much-ram digitaltrends]

What resources when

2025-10-24T15:11:54Z

Areinhar: Mention Leviathans

To try to answer the common question “what resources do I have access to and when should I use them”, below we present to approaches to answering the question. The first of which focuses on what resources exist and what they should be used for and the second tries to answer resource needs / current computing problems with resources to fit those needs/ solve those problems.

After you’re found the resource you think you’d like to use (or even when you’re still deciding), we strongly encourage you to attend an computing office hour to help discuss all the things you should be thinking about and get you on the right track quickly.

== What computing resources do you have access to ==

=== Your personal computer ===

Your personal computer is great for small jobs that take less than 30 minutes, uses few cores and not too much RAM.

'''Pro:'''

#Naturally the easiest tool to use since it’s yours. Best for short jobs that require only a few cores.

'''Cons:'''

#
capacity:
*few cores: 4-8 cores common on laptops<ref><p>From [http://laptopmag.com/articles/Cpu-comparison laptopmag]</p></ref> (hyperthreading means 8-16 “cores”).
*low RAM: 4-32 GB of common on laptops<ref><p>From [http://digitaltrends.com/computing/how-much-ram digitaltrends]</p></ref>, used across all cores
*“no” GPUs: you don’t have GPUs that are optimized for Deep Learning

#
other usage: You often want to use your computer for other things (and it’s built for that exact thing). It’s possible you or someone you know has has had to leave their laptop at home for the day/ run code overnight. When they need to run something intensive -OR- long. But you need that laptop for other work / homework…

The below XKCD comic doesn’t have to be you… (though breaks are good):

[[File:Compiling.png|thumb|none|https://xkcd.com/303/]]

'''Conclusion:''' Your personal computer isn’t the best for when doing a lot of computing and its easy to put jobs elsewhere!

'''Common use-cases'''

#Most class projects
#Non-data problems
#Non-complex structure problems

'''Associated Articles''':

*[[Parallel_computing|parallelization]]

=== The department servers ===

The departmental servers should be seen as a “sandbox”. You can experiment with lots of things on the servers and it doesn’t cost you much to do. The Leviathans and Hydras have more RAM than a typical laptop. The Leviathans also have top-of-the-line CPUs that should be faster than most laptops, though the Hydras have older CPUs that may not be faster than running code on your own laptop.

'''Pros:'''

#Best for long
#Plenty of RAM per machine (128 GB on Leviathans + hydra7 and 8, 64 GB on other hydras)
#Very low barrier to entry (aka easy to get started)
#File management system shares across all servers

'''Cons:'''

#usage: The department server is a bit of a “wild-west” in terms of usage (it varies how many people use it at a given time)
#cores per server: the number of cores per server is only 12 (Hydras) or 16 (Leviathans)

'''Conclusion:''' Great place to start for more intensive computing.

'''Associated Articles''':

*[[Linux_and_department_servers|Linux and department servers]]
'''Server Monitoring'''

* [https://info.stat.cmu.edu/grafana/d/beuw1q6y3dgjkc/hydra-cluster?orgId=1 Hydra Cluster Usage Monitor]
* [https://info.stat.cmu.edu/grafana/d/asjkldfbnvajdsfh213/leviathan-cluster?orgId=1 Leviathan Cluster Usage Monitor]

== PSC (Pittsburgh supercomputer center) ==

The Pittsburgh supercomputer center provides supercomputers than have massive amounts of cores (but also can find some with large RAMs and GPUs). Generally speaking, the supercomputing resources are often used when you have extremely parallelizable code that would take a very long time if you ran it on just a few cores (e.g. your computer or a few hydras).

'''Pros:'''

#A massive amount of cores and RAM that can be leveraged for extremely parallelizable computing
#Easier to access in the statistics department

'''Cons:'''

#Requires learning of new job management system (though quick to learn - see [Slurm])
#Some options to access the resources can take time or require going through your advisor
#''currently (2021)'' has been having problems with servers crashing

'''Associated Articles''':

*[[PSC|PSC]]
*[[Slurm|Slurm]]

== GPU Computing ==
The Statistics & Data Science Department offers two main GPU computing environments.

'''Ghidorahs'''

* 3 physical servers
* RTX 4090 GPUs
* Connected to the Stats network share
* Your statistics directory account will grant you access

'''Wright'''

* For a detail overview, please check [[GPU Cluster|here]]
* You must request access
* The Statistics network share is not accessible

= Computing resource flowchart =

These flow charts try to help capture what’s available and when you might use them.

Visit the links below to view current computing resource usage. Select a server with the least amount of users and lowest resource usage.

* [https://info.stat.cmu.edu/grafana/d/fdkb6l4klj400d/hydra-cluster?orgId=1 Hydra Cluster Usage Monitor]

* [https://info.stat.cmu.edu/grafana/d/edkar1rgy3l6od/ghidorah-cluster?orgId=1 Ghidorah Cluster Usage Monitor]
[[File:ProblemsOnYourPersonalComputerUpdated.png|center|thumb|1000x1000px]]

From [http://laptopmag.com/articles/Cpu-comparison laptopmag]
<br/>      2. <font color="blue">↑</font>
From [http://digitaltrends.com/computing/how-much-ram digitaltrends]

Linux and department servers

2025-10-24T15:07:42Z

Areinhar: Mention Leviathans

The department has two sets of servers, the Leviathans and the Hydras, that can help you run large analyses and gain a better understanding of common command line structure for more high scale computing. One can also view the department servers a bit more as a “sandbox” or “playground” as there are very few constraints and it’s okay to not run as polished code as you might on the [[PSC]] or Azure.

['''Request:''' Windows users (and all users) - if you see comments that are outdated / less helpful than expected please contribute.]

== What resources exist ==

# The department provides 13 hydra servers (named {<code>hydra1</code>, ... , <code>hydra13</code>}<code>.stat.cmu.edu</code> and 6 leviathan servers (named {<code>leviathan1</code>, ..., <code>leviathan6</code>}<code>.stat.cmu.edu</code>)
# Each Hydra has 12 CPU cores and 64 GB of RAM.
## The exceptions are hydra7 and hydra8, which have 128 GB of RAM. Use these if you have particularly large jobs that need lots of memory.
# Each Leviathan has 16 CPU cores and 128 GB of RAM. The Leviathan CPUs are much faster than the Hydra CPUs.
# The Hydras and Leviathans all share the same data file system. This allows you to upload a file to any of the servers and then have it on all of them.
# [[OpenMPI]] is installed
#* This allows MPI based parallelization to be used
#* Note that the hydras ARE NOT network connected, so MPI can only be used within a single hydra. It cannot communicate across multiple hydras.
#* If you need to run something in parallel across multiple nodes, you should use [[PSC]] instead.

''See [[what_resources_when| What resources when]] for guides to help decide which resource you should be using.''

== Using the servers respectfully ==
Because the servers are a shared department resource it is important to use them respectfully. There are many servers and multiple cores (CPUs) per server. To be respectful of other users:

# '''Check the current usage''' of the server you're on before using it.
# Do not use '''all''' the cores on a server, and do not use '''all''' the servers at the same time, so others can get work done.
# If you have a long-running job, '''deprioritize it''' so others can run too.

Each of these is easy to check.

=== Checking current usage ===
To check which servers are currently being used and which are open, see the [https://info.stat.cmu.edu/grafana/d/beuw1q6y3dgjkc/hydra-cluster?orgId=1 Grafana dashboard for the hydras] and [https://info.stat.cmu.edu/grafana/d/asjkldfbnvajdsfh213/leviathan-cluster?orgId=1 for the Leviathans]. It lists them all and their current usage.

When you're logged in to one node, you can use the <code>top</code> command. <code>top</code> displays the current usage of the server’s resources with a list of jobs run. It also tells you the percentage of the cores that are being used and the amount of RAM (memory) that is available. See [https://www.howtogeek.com/668986/how-to-use-the-linux-top-command-and-understand-its-output/ howtogeek] resources on how to interpret it all (the documentation is very clean, so we encourage you to read up ther

=== Limiting the cores you use ===
We recommend that when using parallel processing, use fewer than the total number of cores on the server. Most parallel processing packages let you choose the number of cores to use; the hydras have 12, so pick something smaller.

=== Deprioritize when possible ===
<code>nice/renice</code> allows you to lower the priority of your jobs, so the operating system lets other processes run first. See [https://www.howtogeek.com/411979/how-to-set-process-priorities-with-the-nice-and-renice-commands-in-linux/ howtogeek] resources on the basics. <code>nice</code> sets the priority, and <code>renice</code> updates your priority. On the department servers, everyone starts at the same priority, and you can only '''deprioritize''' your jobs -- you can't make them higher priority than everyone else.<syntaxhighlight lang="bash">nice -20 Rscript some-big-script.R
renice -n 20 processid # get the process ID from top</syntaxhighlight>
== Introduction to connecting to the server and useful Linux commands ==

The department servers all run Linux, and require you to use a command line to interact with it (as compared to Rstudio and other GUIs).
=== Getting Access: (<code>ssh</code>) ===

To get onto the server and type code you’ll need to access it through <code>ssh</code>.

<ul>
<li>For Macs (and Linux), open up a <code>terminal</code> and type
<syntaxhighlight lang="bash">ssh andrew_id@hydra1.stat.cmu.edu</syntaxhighlight>

you will then have to type in your password (if you don’t know it or don’t know it or have never accessed the server before - email [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu]).
</li>
<li>
For Windows, there's a few options (with different pros and cons). [https://www.makeuseof.com/tag/4-easy-ways-to-use-ssh-in-windows/ Makeuseof] provides an overview and introduction to how to get started on a few approaches, including PuTTY and Windows PowerShell.

Again, make sure to email [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu] to check that you have an account / or what your password is if you forgot.
</li>
</ul>
If you’re accessing the servers a lot, you might also look into aliases and password storing for the <code>ssh</code> commands. - Mac/Linux: [https://scotch.io/tutorials/how-to-create-an-ssh-shortcut scotch.io] - Windows: Powershell: [https://docs.microsoft.com/en-us/windows/terminal/tutorials/ssh microsoft], PuTTY: [https://www.makeuseof.com/tag/4-easy-ways-to-use-ssh-in-windows/ makeuseof]

=== Passwordless ssh ===
To get on the server without typing in the password, you need to first generate a SSH key pair. This creates a private key that identifies your laptop, and a public key that the servers can use to recognize your private key.

* For Linux or Mac, open up a terminal and check whether you already have a key pair: run <code>ls ~/.ssh</code> and see if you have any files with names starting <code>id_</code>. If you do, you already have a key pair, and you can skip to the <code>ssh-copy-id</code> step below.
* Next, generate a key pair (both public and private key). There are different types of keys (RSA, ECDSA, and Ed25519). The <code>-t</code> option specifies the type of key pair.
ssh-keygen -t ed25519
* A prompt for what to name the key pair and what folder to place it in should appear. The username should be the username for your local computer and <code>id_ed25519</code> is the key pair name. You can change it. The local key should be stored in the <code>~/.ssh</code> folder on your local computer.
Generating public/private ed25519 key pair.
Enter file in which to save the key (/home/username/.ssh/id_ed25519):
* Hit Enter to accept the default name and location, or enter the folder and name manually.
* A prompt for a passphrase should appear. This is for securing the local key (on local computer) and is optional. <code>Enter Passphrase (empty for no passphrase):</code>

* Make sure there is a <code>~/.ssh</code> folder on the server. If not, make a .ssh folder.
* Next, upload the public key to remote server. This command will upload the public key to all the hydras and ghidorahs since they share a file system.
ssh-copy-id andrew_id@hydra1.stat.cmu.edu
* To ensure the permissions of the <code>~/.ssh</code> directory are updated, log into the hydras and run:
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
You should now be able to SSH into the server without being prompted for a password.

=== File sharing (<code>scp</code> +) ===

There is also command line code to copy files from your personal computer to the server. This is done with a SCP (secure copy) tool. There are also GUIs (graphical user interfaces) that allow you to “drag and drop” files from your computer to the server (and visa-versa). Currently, a lot of students use GUIs to deal with file sharing.

==== Command line <code>scp</code>====

You can file share either by using <code>scp</code>, for example

<ol>
<li> Mac/Linux: This command is very similar to <code>cp</code> (the file copier).
<ol>
<li>Copy from server
<syntaxhighlight lang="bash"> scp username@from_host:path/to/file.txt /local/directory/ </syntaxhighlight>
<syntaxhighlight lang="bash">scp -r username@from_host:path/to/folder/ local/directory/</syntaxhighlight>
</li>
<li>Copy to server
 
<syntaxhighlight lang="bash"> scp local/file.txt username@to_host:/remote/directory/</syntaxhighlight>
<syntaxhighlight lang="bash"> scp -r local/directory/ username@to_host:/remote/directory/</syntaxhighlight>
</li>
</ol>

<code>scp</code> references: [https://haydenjames.io/linux-securely-copy-files-using-scp/ haydenjames.io]

</li>
<li>Windows: Again, this process is a bit more complicated.
<ol>
<li>A PuTTY + Command Prompt provides a version that is very similar to Mac/Linux <code>scp</code> (reference: [https://success.tanaza.com/s/article/How-to-use-SCP-command-on-Windows tanaza]), and requires minor changes in the call  (<code>pscp -scp</code> instead of <code>scp)</code>. </li>
<li>A Powershell specific command is a bit more complicated / not similar to Mac/Linux <code>scp</code> (reference: [https://www.nsoftware.com/kb/articles/psserver-scp.rst nsoftware]) </li>
</ol>
</li>
</ol>

===  GUI “drag-and-drop” file sharing ===
A '''lot''' of students use '''GUI''' interfaces to transfer files instead of <code>scp</code> command line. These tools allow you to “drag and drop” files from one location to another after the connection is set up.

Some examples include:

<ol>
<li>Mac:
<ol><li>
[https://cyberduck.io/ Cyberduck]
</li></ol></li>
<li>Linux:
<ol><li>
[https://www.openssh.com/ OpenSSH]
</li></ol></li>
<li> Windows: lots of options - [https://www.thegeekstuff.com/2011/06/windows-sftp-scp-clients/ thegeekstuff]
<ol>
<li>[https://www.putty.org/ PuTTY] </li>
<li>[https://winscp.net/eng/download.php WinSCP] </li></ol>
</li>
</ol>
=== Basic CLI (Linux command line): <code>bash</code>===

Understanding the basic Linux command line is important to be able to navigate when you are actually on the server. Below are some based commands to move around the file system, manipulate files, and view storage structure.

{| class="wikitable"
|-
! command
! translation
|-
| <code>pwd</code>
| print the current working directory
|-
| <code>cd _dir_</code>
| change directory
|-
| <code>rm _file_</code>
| remove / delete file
|-
| <code>mv _file_ _dir_</code>
| move file (can change name)
|-
| <code>cp _file_ _dir_</code>
| copy file
|-
| <code>ls _dir_</code>
| list files in directory
|}

use <code>man _command_</code> to get a listing / manual of the command.

Additional resources that capture more information on the command line and are still geared to statistical computing can be found here: [http://www.jarrodmillman.com/rcsds/standard/bash.html UCB 2015 reproducible DS lecture notes].

'''TODO: add information about <code>sed</code>, <code>chmod</code>, and a link to bash files.'''

== Text editors ==

There are multiple options to view code on the server side. They include:

*vim
*… [to be added later]

== Running jobs ==

We have included thoughts about running jobs in two places.

#The first looks at tools that allow you to keep running jobs on the server when you close the connection to the server (turn off your laptop, etc.). These tools are call [[Terminal_multiplexers|terminal multiplexers]]. 
#The second set of tools focus on how to run your [[R_on_the_server|R analysis]] or [[Python_on_the_server|Python analysis]] code on the server.

== Server specs ==

Generally speaking, our servers have multiple CPU cores, but no GPU cores. For deep learning tasks that need GPUs, consider using the [[GPU Cluster|GPU cluster]].

To examine the specs yourself, use
<syntaxhighlight lang="bash">cat /proc/meminfo
cat /proc/cpuinfo</syntaxhighlight>

to see some information.

Department Servers

2025-10-24T14:46:50Z

Areinhar: /* Hydras */ mention which hydras have more RAM

== Hydras ==
The Hydras are virtual based serves hosted in the Campus Cloud. They offer slightly higher stability and are better for smaller computing jobs.

Hardware Specifications

* Intel Xeon CPU
** 12 cores @2.6Ghz
* 64GB of RAM (128GB on hydra7 and hydra8)
* NAS used for /home storage

[https://info.stat.cmu.edu/grafana/d/beuw1q6y3dgjkc/hydra-cluster?orgId=1 Grafana Monitoring]
== Ghidorahs ==
The Ghidorahs are physical servers located throughout Baker Hall. They are designed for lighter GPU jobs.

Hardware Specifications

* Core i9-13900KF CPU
** 8 performance cores @5.4Ghz
** 16 Efficiency cores @4.3Ghz
* 64GB of RAM
* RTX 4090 24GB of video memory

[https://info.stat.cmu.edu/grafana/d/edkar1rgy3l6od/ghidorah-cluster?orgId=1 Grafana Monitoring]
== Leviathans ==
The Leviathans are physical servers located throughout Baker Hall. They are designed for heavy CPU and RAM jobs.

Hardware Specifications

* AMD Ryzen 9 9950x
** 16 cores @5.7Ghz
* 128GB of RAM
* 4TB of local NVME storage
** Mounted at /scratch
** Data stored in this location will only be accessible on the original server.

[https://info.stat.cmu.edu/grafana/d/asjkldfbnvajdsfh213/leviathan-cluster?orgId=1 Grafana Monitor]

== Storage ==
Each server's "/home" directory located on the department's NAS. This means your files will be accessible on every server.

The NAS is a total of 100TB. This space is shared and monitored for fair usage. If you store large amounts of data, you may be asked if it is all necessary.

To see available space on the NAS, run the command below in a terminal on the server.<syntaxhighlight lang="bash">
df -h | grep "/home"
</syntaxhighlight>To see your total usage run the command below. ''It may take a few minutes to finish''<syntaxhighlight lang="bash">
du -sh ~
</syntaxhighlight>

== Using the Servers ==

Accessing the servers requires using an SSH compatible terminal. Applications such as "Command Prompt" on Windows or "Terminal" on MacOS can achieve this.
To SSH to a server, open either app listed previously and enter the SSH command shown below. Be sure to use your StatDS computing account.<syntaxhighlight lang="bash">
ssh statdsusername@hydra1.stat.cmu.edu
</syntaxhighlight>In the line above, replace "hydra1" with the server you want to connect to.
{| class="wikitable"
|+
!Hydras
!Ghidorahs
!Leviathans
|-
|hydra1
|ghidroah1
|leviathan1
|-
|hydra2
|ghidorah2
|leviathan2
|-
|hydra3
|ghidorah3
|leviathan3
|-
|hydra4
|
|leviathan4
|-
|hydra5
|
|leviathan5
|-
|hydra6
|
|leviathan6
|-
|hydra7
|
|
|-
|hydra8
|
|
|-
|hydra9
|
|
|-
|hydra10
|
|
|-
|hydra11
|
|
|-
|hydra12
|
|
|-
|hydra12
|
|
|}

== Software ==
The servers within Statistics & Data Science offer a variety of software. Sudo access, also referred to as the ability to install software, on department servers is not allowed. If a software package you would like to use is not listed below, please contact remarks@stat.cmu.edu.
{| class="wikitable"
|+
!
!Hydras
!Ghidorahs
!Leviathans
|-
|Python
|X
|X
|X
|-
|R
|X
|X
|X
|-
|Apptainer
|X
|X
|X
|-
|Quarto
|X
|
|
|-
|Gurobi
|X
|
|
|-
|Rstudio-Server
|X
|
|X
|-
|Matlab
|X
|
|X
|-
|Julia
|X
|
|X
|}

Main Page

2025-09-18T17:24:35Z

Areinhar: /* Managing servers and environments */ fix internal link

Anyone can read this wiki.  If you want to write/edit stuff in the wiki, you will need to log in with your account provided by the Statistics & Data Science Department computing group. This Wiki is intended to be a repository for instruction, information and ideas related to the many computing environments available to members and affiliates of the Statistics & Data Science Department.

== Getting to know the computing resources & getting support ==

If you're interested in knowing about what resources you have access to (and/or) what resources you have access to that might solve a problem you're having, please visit our [[What_resources_when|"what resources when" article]].

To get support that is beyond this website please attend office hours below and/or email [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu] for support (for more details about how the support ticketing works through [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu] + ServiceNow please see the [[Service_now|ServiceNow]] article).

Reset your StatDS password here - https://util.stat.cmu.edu/passwordreset/

== Department Contacts ==

• [mailto:events@stat.cmu.edu events@stat.cmu.edu]: seminars, conferences & workshops, and calendar updates

• [mailto:news@stat.cmu.edu news@stat.cmu.edu]: dept/dept member-related public news & announcements

• [mailto:lists@stat.cmu.edu lists@stat.cmu.edu]: department email lists

• [mailto:webmaster@stat.cmu.edu webmaster@stat.cmu.edu]: website updates

• [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu]: computer/tech requests

• [mailto:rooms@stat.cmu.edu rooms@stat.cmu.edu]: request space for meetings and TA office hours

• [mailto:reimbursements@stat.cmu.edu reimbursements@stat.cmu.edu]: travel and reimbursement requests

== Computing Office Hours ==

These office hour opportunities provide a dedicated time for you to come ask questions about research computing.  These include code advice, how to access certain resources, what the best resources are for your project, or to learn about new connections to help you progress in your education and research. Staff Office hour times will be sent out each week on Monday as a calendar invite; accept any of the invitations to add it to your personal calendar. PhD TA Office Hours are below, unless a different time is communicated.

#PhD TA Office Hours for the first half of summer 2025 are held by Elizabeth Cucuzzella in FMS Room 320 on Mondays from 2:30 pm to 4:30 pm. PhD students can also message on the #computing channel on slack.
#Staff Office Hours are held by appointment by Carl Skipper in BH 129G. Schedule via [mailto:lists@stat.cmu.edu email].

See [[Computing_office_hour_usage|computing office hour usage]] for what office hours can be used for.

== Computing Wiki Pages ==

=== Managing servers and environments ===

==== Introductory Tutorials ====
[[Basic Linux Commands|Introduction to Basic Linux Commands]]

==== Virtual environments ====
*[[Virtual Environment Introduction|Why use virtual environments?]]
*R's [[Renv]]
*Python's [[Python_virtual_environments|virtual environments (2 options)]]
*Old instructions for [[Conda with Python (Reference Only)|conda with Python]]

==== Containers ====
*[[Introduction to containers]]
*[[Docker, and why-you-should-use-it-for-your-research]]
*[[Tutorials|Basic tutorial]]
*[[Running Docker locally and on Hydras]]

=== Department computing resources ===

==== Printing ====
*[[Department_Printers|Department Printers]]

==== Web Hosting ====
#[[How to host your webpage under stat.cmu.edu]]
#[[Bulding your website with Hugo]]

==== Servers ====
*[[Department Servers]].
*#[[Linux and department servers|Using the Servers Overview]] -This wiki introduces you to the '''hydra''' servers, basic Linux commands, and more.
*#[[Terminal_multiplexers|Terminal multiplexers (screen and tmux)]]. This wiki presents tools for use on the ''hydra'' that can allow you to run jobs even when your not on the server, and much much more!
*#[[Python on the server]]. This wiki introduces uses command line approaches to running <code>python</code> on a virtual machine, and how to use jupyter notebooks through the server as well.
*#[[R on the server]]. This wiki introduces uses command line approaches to running <code>R</code> on a virtual machine
*#[https://info.stat.cmu.edu/index.php?title=Department_Servers#Software Software] - overview of the software installed and ready for use.
*#[https://info.stat.cmu.edu/index.php?title=Monitoring_with_Grafana Monitor Server Usage]

*[[Department GPU Servers]], for small jobs needing 1 GPU.
*[[GPU Cluster]], for larger jobs that may need multiple GPUs and many CPUs.
*[[PSC]], the Pittsburgh Supercomputing Center. Free accounts and usage credits are available
*#[[ACCESS]]
*#[[Slurm]]. This wiki introduces the job management system uses at PSC.
*#[[Parallelization_on_PSC/Bridges-2]]
*#[[Access PSC compute nodes directly from VSCode]]
*[[Open OnDemand]]

=== Development guides ===

====[[Parallel_computing|Parallelization]]====
*[[Parallelization_in_r|R-specific parallelization]]
*[[Parallelization_in_python|Python-specific parallelization]]

==== Package development ====
*[[Developing_R_packages|R package development]]
*[[Developing_python_packages|Python package development]]
[[Terminal automation]]. This wiki introduces approaches to automate Linux/OS command line calls.
*[[Bash files]]. This wiki presents potential uses of Bash files including automated file/folder creation, parallelization and much more!
*[[Make]]. This wiki presents potential uses of <code>make</code> and Makefiles including automated workflow and dependency tracking
*[[Jumping to frequently used folders and automating environment setups]]

==== Version Control ====
See the guide provided in the site of [https://36-750.github.io/tools/version-control/ 36750 - Statistical Computing]

==== Reference management ====
#Zotero
#Mendeley

==== Integrated Development Environments (IDEs) ====
#[[VSCode|VSCode]]
#PyCharm
#RStudio
#Eclipse

==== Text Editors ====
#Vim
#Nano
#Emacs
#Atom
#Sublime

See also the guide provided in the site of [https://36-750.github.io/tools/editors/ 36750 - Statistical Computing]

==== AI Tools ====
* [[ChatGPT and other LLMs]]

*[[github_copilot|GitHub Copilot]]

==== Miscellaneous====
#[[Windows_Subsystem_for_Linux_(WSL)_and_Python|Windows Subsystem for Linux (WSL) and Python]]
#[[Amphetamine|Amphetamine: How to keep your computer awake]]

===General Department Computing Resources===

*Fall 2021 [[Computing_seminar_resources|Computing Seminar Resources]] (video and slides)

==Ideas for the Wiki==

Please see the [[TODO|TODO page]] to post suggestions and errata.

==Using the Wiki==

*Here is some [[Generic_Wiki_Information|Generic Wiki Information]] that you may be interested in, such as a users' guide and other resources for Wiki users.

== Archived Pages ==
These area pages that are potentially useful for some users, but no longer reflect the computing resources of the department writ large.

Azure
#[[Azure_getting_started|Azure: getting started]]. This wiki walks you through the steps to create a virtual machine through Azure
#[[How to obtain a virtual machine with a GPU]]
#[[Attach a disk to a virtual machine]]

Wright GPU Cluster

2025-08-30T14:49:57Z

Areinhar: /* Submitting batch jobs */ adjust default email config to email on error too

Linux and department servers

2025-06-12T16:15:13Z

Areinhar: /* Passwordless ssh */

The department has a set of servers, the hydras, that can help you run large analyses and gain a better understanding of common command line structure for more high scale computing. One can also view the department servers a bit more as a “sandbox” or “playground” as there are very few constraints and it’s okay to not run as polished code as you might on the [[PSC]] or Azure.

['''Request:''' Windows users (and all users) - if you see comments that are outdated / less helpful than expected please contribute.]

== What resources exist ==

# The department provides 13 servers (named {<code>hydra1</code>, ... , <code>hydra13</code>}<code>.stat.cmu.edu</code>.
# Each hydra has 12 CPU cores, and 64 GB of RAM (per core).
## The exceptions are hydra7 and hydra8, which have 128 GB of RAM. Use these if you have particularly large jobs that need lots of memory.
# The hydras all share the same data file system. This allows you to upload a file to any of the hydras and then have it on all of them.
# [[OpenMPI]] is installed
#* This allows MPI based parallelization to be used
#* Note that the hydras ARE NOT network connected, so MPI can only be used within a single hydra. It cannot communicate across multiple hydras.
#* If you need to run something in parallel across multiple nodes, you should use [[PSC]] instead.

''See [[what_resources_when| What resources when]] for guides to help decide which resource you should be using.''

== Using the servers respectfully ==
Because the servers are a shared department resource it is important to use them respectfully. There are many servers and multiple cores (CPUs) per server. To be respectful of other users:

# '''Check the current usage''' of the server you're on before using it.
# Do not use '''all''' the cores on a server, and do not use '''all''' the servers at the same time, so others can get work done.
# If you have a long-running job, '''deprioritize it''' so others can run too.

Each of these is easy to check.

=== Checking current usage ===
To check which servers are currently being used and which are open, see the [https://info.stat.cmu.edu/grafana/d/fdkb6l4klj400d/hydra-cluster?orgId=1 Grafana dashboard for the hydras]. It lists them all and their current usage.

When you're logged in to one node, you can use the <code>top</code> command. <code>top</code> displays the current usage of the server’s resources with a list of jobs run. It also tells you the percentage of the cores that are being used and the amount of RAM (memory) that is available. See [https://www.howtogeek.com/668986/how-to-use-the-linux-top-command-and-understand-its-output/ howtogeek] resources on how to interpret it all (the documentation is very clean, so we encourage you to read up ther

=== Limiting the cores you use ===
We recommend that when using parallel processing, use fewer than the total number of cores on the server. Most parallel processing packages let you choose the number of cores to use; the hydras have 12, so pick something smaller.

=== Deprioritize when possible ===
<code>nice/renice</code> allows you to lower the priority of your jobs, so the operating system lets other processes run first. See [https://www.howtogeek.com/411979/how-to-set-process-priorities-with-the-nice-and-renice-commands-in-linux/ howtogeek] resources on the basics. <code>nice</code> sets the priority, and <code>renice</code> updates your priority. On the department servers, everyone starts at the same priority, and you can only '''deprioritize''' your jobs -- you can't make them higher priority than everyone else.<syntaxhighlight lang="bash">nice -20 Rscript some-big-script.R
renice -n 20 processid # get the process ID from top</syntaxhighlight>
== Introduction to connecting to the server and useful Linux commands ==

The department servers all run Linux, and require you to use a command line to interact with it (as compared to Rstudio and other GUIs).
=== Getting Access: (<code>ssh</code>) ===

To get onto the server and type code you’ll need to access it through <code>ssh</code>.

<ul>
<li>For Macs (and Linux), open up a <code>terminal</code> and type
<syntaxhighlight lang="bash">ssh andrew_id@hydra1.stat.cmu.edu</syntaxhighlight>

you will then have to type in your password (if you don’t know it or don’t know it or have never accessed the server before - email [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu]).
</li>
<li>
For Windows, there's a few options (with different pros and cons). [https://www.makeuseof.com/tag/4-easy-ways-to-use-ssh-in-windows/ Makeuseof] provides an overview and introduction to how to get started on a few approaches, including PuTTY and Windows PowerShell.

Again, make sure to email [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu] to check that you have an account / or what your password is if you forgot.
</li>
</ul>
If you’re accessing the servers a lot, you might also look into aliases and password storing for the <code>ssh</code> commands. - Mac/Linux: [https://scotch.io/tutorials/how-to-create-an-ssh-shortcut scotch.io] - Windows: Powershell: [https://docs.microsoft.com/en-us/windows/terminal/tutorials/ssh microsoft], PuTTY: [https://www.makeuseof.com/tag/4-easy-ways-to-use-ssh-in-windows/ makeuseof]

=== Passwordless ssh ===
To get on the server without typing in the password, you need to first generate a SSH key pair. This creates a private key that identifies your laptop, and a public key that the servers can use to recognize your private key.

* For Linux or Mac, open up a terminal and check whether you already have a key pair: run <code>ls ~/.ssh</code> and see if you have any files with names starting <code>id_</code>. If you do, you already have a key pair, and you can skip to the <code>ssh-copy-id</code> step below.
* Next, generate a key pair (both public and private key). There are different types of keys (RSA, ECDSA, and Ed25519). The <code>-t</code> option specifies the type of key pair.
ssh-keygen -t ed25519
* A prompt for what to name the key pair and what folder to place it in should appear. The username should be the username for your local computer and <code>id_ed25519</code> is the key pair name. You can change it. The local key should be stored in the <code>~/.ssh</code> folder on your local computer.
Generating public/private ed25519 key pair.
Enter file in which to save the key (/home/username/.ssh/id_ed25519):
* Hit Enter to accept the default name and location, or enter the folder and name manually.
* A prompt for a passphrase should appear. This is for securing the local key (on local computer) and is optional. <code>Enter Passphrase (empty for no passphrase):</code>

* Make sure there is a <code>~/.ssh</code> folder on the server. If not, make a .ssh folder.
* Next, upload the public key to remote server. This command will upload the public key to all the hydras and ghidorahs since they share a file system.
ssh-copy-id andrew_id@hydra1.stat.cmu.edu
* To ensure the permissions of the <code>~/.ssh</code> directory are updated, log into the hydras and run:
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
You should now be able to SSH into the server without being prompted for a password.

=== File sharing (<code>scp</code> +) ===

There is also command line code to copy files from your personal computer to the server. This is done with a SCP (secure copy) tool. There are also GUIs (graphical user interfaces) that allow you to “drag and drop” files from your computer to the server (and visa-versa). Currently, a lot of students use GUIs to deal with file sharing.

==== Command line <code>scp</code>====

You can file share either by using <code>scp</code>, for example

<ol>
<li> Mac/Linux: This command is very similar to <code>cp</code> (the file copier).
<ol>
<li>Copy from server
<syntaxhighlight lang="bash"> scp username@from_host:path/to/file.txt /local/directory/ </syntaxhighlight>
<syntaxhighlight lang="bash">scp -r username@from_host:path/to/folder/ local/directory/</syntaxhighlight>
</li>
<li>Copy to server
 
<syntaxhighlight lang="bash"> scp local/file.txt username@to_host:/remote/directory/</syntaxhighlight>
<syntaxhighlight lang="bash"> scp -r local/directory/ username@to_host:/remote/directory/</syntaxhighlight>
</li>
</ol>

<code>scp</code> references: [https://haydenjames.io/linux-securely-copy-files-using-scp/ haydenjames.io]

</li>
<li>Windows: Again, this process is a bit more complicated.
<ol>
<li>A PuTTY + Command Prompt provides a version that is very similar to Mac/Linux <code>scp</code> (reference: [https://success.tanaza.com/s/article/How-to-use-SCP-command-on-Windows tanaza]), and requires minor changes in the call  (<code>pscp -scp</code> instead of <code>scp)</code>. </li>
<li>A Powershell specific command is a bit more complicated / not similar to Mac/Linux <code>scp</code> (reference: [https://www.nsoftware.com/kb/articles/psserver-scp.rst nsoftware]) </li>
</ol>
</li>
</ol>

===  GUI “drag-and-drop” file sharing ===
A '''lot''' of students use '''GUI''' interfaces to transfer files instead of <code>scp</code> command line. These tools allow you to “drag and drop” files from one location to another after the connection is set up.

Some examples include:

<ol>
<li>Mac:
<ol><li>
[https://cyberduck.io/ Cyberduck]
</li></ol></li>
<li>Linux:
<ol><li>
[https://www.openssh.com/ OpenSSH]
</li></ol></li>
<li> Windows: lots of options - [https://www.thegeekstuff.com/2011/06/windows-sftp-scp-clients/ thegeekstuff]
<ol>
<li>[https://www.putty.org/ PuTTY] </li>
<li>[https://winscp.net/eng/download.php WinSCP] </li></ol>
</li>
</ol>
=== Basic CLI (Linux command line): <code>bash</code>===

Understanding the basic Linux command line is important to be able to navigate when you are actually on the server. Below are some based commands to move around the file system, manipulate files, and view storage structure.

{| class="wikitable"
|-
! command
! translation
|-
| <code>pwd</code>
| print the current working directory
|-
| <code>cd _dir_</code>
| change directory
|-
| <code>rm _file_</code>
| remove / delete file
|-
| <code>mv _file_ _dir_</code>
| move file (can change name)
|-
| <code>cp _file_ _dir_</code>
| copy file
|-
| <code>ls _dir_</code>
| list files in directory
|}

use <code>man _command_</code> to get a listing / manual of the command.

Additional resources that capture more information on the command line and are still geared to statistical computing can be found here: [http://www.jarrodmillman.com/rcsds/standard/bash.html UCB 2015 reproducible DS lecture notes].

'''TODO: add information about <code>sed</code>, <code>chmod</code>, and a link to bash files.'''

== Text editors ==

There are multiple options to view code on the server side. They include:

*vim
*… [to be added later]

== Running jobs ==

We have included thoughts about running jobs in two places.

#The first looks at tools that allow you to keep running jobs on the server when you close the connection to the server (turn off your laptop, etc.). These tools are call [[Terminal_multiplexers|terminal multiplexers]]. 
#The second set of tools focus on how to run your [[R_on_the_server|R analysis]] or [[Python_on_the_server|Python analysis]] code on the server.

== Server specs ==

Generally speaking, our servers have multiple CPU cores, but no GPU cores. For deep learning tasks that need GPUs, consider using the [[GPU Cluster|GPU cluster]].

To examine the specs yourself, use
<syntaxhighlight lang="bash">cat /proc/meminfo
cat /proc/cpuinfo</syntaxhighlight>

to see some information.

Linux and department servers

2025-06-12T15:37:04Z

Areinhar: /* Passwordless ssh */

The department has a set of servers, the hydras, that can help you run large analyses and gain a better understanding of common command line structure for more high scale computing. One can also view the department servers a bit more as a “sandbox” or “playground” as there are very few constraints and it’s okay to not run as polished code as you might on the [[PSC]] or Azure.

['''Request:''' Windows users (and all users) - if you see comments that are outdated / less helpful than expected please contribute.]

== What resources exist ==

# The department provides 13 servers (named {<code>hydra1</code>, ... , <code>hydra13</code>}<code>.stat.cmu.edu</code>.
# Each hydra has 12 CPU cores, and 64 GB of RAM (per core).
## The exceptions are hydra7 and hydra8, which have 128 GB of RAM. Use these if you have particularly large jobs that need lots of memory.
# The hydras all share the same data file system. This allows you to upload a file to any of the hydras and then have it on all of them.
# [[OpenMPI]] is installed
#* This allows MPI based parallelization to be used
#* Note that the hydras ARE NOT network connected, so MPI can only be used within a single hydra. It cannot communicate across multiple hydras.
#* If you need to run something in parallel across multiple nodes, you should use [[PSC]] instead.

''See [[what_resources_when| What resources when]] for guides to help decide which resource you should be using.''

== Using the servers respectfully ==
Because the servers are a shared department resource it is important to use them respectfully. There are many servers and multiple cores (CPUs) per server. To be respectful of other users:

# '''Check the current usage''' of the server you're on before using it.
# Do not use '''all''' the cores on a server, and do not use '''all''' the servers at the same time, so others can get work done.
# If you have a long-running job, '''deprioritize it''' so others can run too.

Each of these is easy to check.

=== Checking current usage ===
To check which servers are currently being used and which are open, see the [https://info.stat.cmu.edu/grafana/d/fdkb6l4klj400d/hydra-cluster?orgId=1 Grafana dashboard for the hydras]. It lists them all and their current usage.

When you're logged in to one node, you can use the <code>top</code> command. <code>top</code> displays the current usage of the server’s resources with a list of jobs run. It also tells you the percentage of the cores that are being used and the amount of RAM (memory) that is available. See [https://www.howtogeek.com/668986/how-to-use-the-linux-top-command-and-understand-its-output/ howtogeek] resources on how to interpret it all (the documentation is very clean, so we encourage you to read up ther

=== Limiting the cores you use ===
We recommend that when using parallel processing, use fewer than the total number of cores on the server. Most parallel processing packages let you choose the number of cores to use; the hydras have 12, so pick something smaller.

=== Deprioritize when possible ===
<code>nice/renice</code> allows you to lower the priority of your jobs, so the operating system lets other processes run first. See [https://www.howtogeek.com/411979/how-to-set-process-priorities-with-the-nice-and-renice-commands-in-linux/ howtogeek] resources on the basics. <code>nice</code> sets the priority, and <code>renice</code> updates your priority. On the department servers, everyone starts at the same priority, and you can only '''deprioritize''' your jobs -- you can't make them higher priority than everyone else.<syntaxhighlight lang="bash">nice -20 Rscript some-big-script.R
renice -n 20 processid # get the process ID from top</syntaxhighlight>
== Introduction to connecting to the server and useful Linux commands ==

The department servers all run Linux, and require you to use a command line to interact with it (as compared to Rstudio and other GUIs).
=== Getting Access: (<code>ssh</code>) ===

To get onto the server and type code you’ll need to access it through <code>ssh</code>.

<ul>
<li>For Macs (and Linux), open up a <code>terminal</code> and type
<syntaxhighlight lang="bash">ssh andrew_id@hydra1.stat.cmu.edu</syntaxhighlight>

you will then have to type in your password (if you don’t know it or don’t know it or have never accessed the server before - email [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu]).
</li>
<li>
For Windows, there's a few options (with different pros and cons). [https://www.makeuseof.com/tag/4-easy-ways-to-use-ssh-in-windows/ Makeuseof] provides an overview and introduction to how to get started on a few approaches, including PuTTY and Windows PowerShell.

Again, make sure to email [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu] to check that you have an account / or what your password is if you forgot.
</li>
</ul>
If you’re accessing the servers a lot, you might also look into aliases and password storing for the <code>ssh</code> commands. - Mac/Linux: [https://scotch.io/tutorials/how-to-create-an-ssh-shortcut scotch.io] - Windows: Powershell: [https://docs.microsoft.com/en-us/windows/terminal/tutorials/ssh microsoft], PuTTY: [https://www.makeuseof.com/tag/4-easy-ways-to-use-ssh-in-windows/ makeuseof]

=== Passwordless ssh ===
To get on the server without typing in the password, you need to first generate a SSH key pair.

* For Linux, open up a terminal and check first that the specific ssh key pair does not exist in the given folder.
* Next, generate a key pair (both public and private key). There are different types of keys (RSA, DSA, ECDSA, and ED25519). The <code>-t</code> option specifies the type of key pair.
ssh-keygen -t rsa
* A prompt for what to name the key pair and what folder to place it in should appear. The username should be the username for your local computer and id_rsa is the key pair name. You can change it. The local key should be stored in the <code>~/.ssh</code> folder on your local computer.
Generating public/private rsa key pair.
Enter file in which to save the key (/home/username/.ssh/id_rsa):
* Hit Enter to accept the default name and location, or enter the folder and name manually.
* A prompt for a passphrase should appear. This is for securing the local key (on local computer) and is optional. <code>Enter Passphrase (empty for no passphrase):</code>

* Make sure there is a <code>~/.ssh</code> folder on the server. If not, make a .ssh folder.
* Next, upload the public key to remote server. This command will upload the public key to all the hydras and ghidorahs since they share a file system.
ssh-copy-id andrew_id@hydra1.stat.cmu.edu
* To ensure the permissions of the <code>~/.ssh</code> directory are updated:
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
You should now be able to SSH into the server without being prompted for a password.

=== File sharing (<code>scp</code> +) ===

There is also command line code to copy files from your personal computer to the server. This is done with a SCP (secure copy) tool. There are also GUIs (graphical user interfaces) that allow you to “drag and drop” files from your computer to the server (and visa-versa). Currently, a lot of students use GUIs to deal with file sharing.

==== Command line <code>scp</code>====

You can file share either by using <code>scp</code>, for example

<ol>
<li> Mac/Linux: This command is very similar to <code>cp</code> (the file copier).
<ol>
<li>Copy from server
<syntaxhighlight lang="bash"> scp username@from_host:path/to/file.txt /local/directory/ </syntaxhighlight>
<syntaxhighlight lang="bash">scp -r username@from_host:path/to/folder/ local/directory/</syntaxhighlight>
</li>
<li>Copy to server
 
<syntaxhighlight lang="bash"> scp local/file.txt username@to_host:/remote/directory/</syntaxhighlight>
<syntaxhighlight lang="bash"> scp -r local/directory/ username@to_host:/remote/directory/</syntaxhighlight>
</li>
</ol>

<code>scp</code> references: [https://haydenjames.io/linux-securely-copy-files-using-scp/ haydenjames.io]

</li>
<li>Windows: Again, this process is a bit more complicated.
<ol>
<li>A PuTTY + Command Prompt provides a version that is very similar to Mac/Linux <code>scp</code> (reference: [https://success.tanaza.com/s/article/How-to-use-SCP-command-on-Windows tanaza]), and requires minor changes in the call  (<code>pscp -scp</code> instead of <code>scp)</code>. </li>
<li>A Powershell specific command is a bit more complicated / not similar to Mac/Linux <code>scp</code> (reference: [https://www.nsoftware.com/kb/articles/psserver-scp.rst nsoftware]) </li>
</ol>
</li>
</ol>

===  GUI “drag-and-drop” file sharing ===
A '''lot''' of students use '''GUI''' interfaces to transfer files instead of <code>scp</code> command line. These tools allow you to “drag and drop” files from one location to another after the connection is set up.

Some examples include:

<ol>
<li>Mac:
<ol><li>
[https://cyberduck.io/ Cyberduck]
</li></ol></li>
<li>Linux:
<ol><li>
[https://www.openssh.com/ OpenSSH]
</li></ol></li>
<li> Windows: lots of options - [https://www.thegeekstuff.com/2011/06/windows-sftp-scp-clients/ thegeekstuff]
<ol>
<li>[https://www.putty.org/ PuTTY] </li>
<li>[https://winscp.net/eng/download.php WinSCP] </li></ol>
</li>
</ol>
=== Basic CLI (Linux command line): <code>bash</code>===

Understanding the basic Linux command line is important to be able to navigate when you are actually on the server. Below are some based commands to move around the file system, manipulate files, and view storage structure.

{| class="wikitable"
|-
! command
! translation
|-
| <code>pwd</code>
| print the current working directory
|-
| <code>cd _dir_</code>
| change directory
|-
| <code>rm _file_</code>
| remove / delete file
|-
| <code>mv _file_ _dir_</code>
| move file (can change name)
|-
| <code>cp _file_ _dir_</code>
| copy file
|-
| <code>ls _dir_</code>
| list files in directory
|}

use <code>man _command_</code> to get a listing / manual of the command.

Additional resources that capture more information on the command line and are still geared to statistical computing can be found here: [http://www.jarrodmillman.com/rcsds/standard/bash.html UCB 2015 reproducible DS lecture notes].

'''TODO: add information about <code>sed</code>, <code>chmod</code>, and a link to bash files.'''

== Text editors ==

There are multiple options to view code on the server side. They include:

*vim
*… [to be added later]

== Running jobs ==

We have included thoughts about running jobs in two places.

#The first looks at tools that allow you to keep running jobs on the server when you close the connection to the server (turn off your laptop, etc.). These tools are call [[Terminal_multiplexers|terminal multiplexers]]. 
#The second set of tools focus on how to run your [[R_on_the_server|R analysis]] or [[Python_on_the_server|Python analysis]] code on the server.

== Server specs ==

Generally speaking, our servers have multiple CPU cores, but no GPU cores. For deep learning tasks that need GPUs, consider using the [[GPU Cluster|GPU cluster]].

To examine the specs yourself, use
<syntaxhighlight lang="bash">cat /proc/meminfo
cat /proc/cpuinfo</syntaxhighlight>

to see some information.

Linux and department servers

2025-06-12T15:35:57Z

Areinhar: /* Passwordless ssh */

The department has a set of servers, the hydras, that can help you run large analyses and gain a better understanding of common command line structure for more high scale computing. One can also view the department servers a bit more as a “sandbox” or “playground” as there are very few constraints and it’s okay to not run as polished code as you might on the [[PSC]] or Azure.

['''Request:''' Windows users (and all users) - if you see comments that are outdated / less helpful than expected please contribute.]

== What resources exist ==

# The department provides 13 servers (named {<code>hydra1</code>, ... , <code>hydra13</code>}<code>.stat.cmu.edu</code>.
# Each hydra has 12 CPU cores, and 64 GB of RAM (per core).
## The exceptions are hydra7 and hydra8, which have 128 GB of RAM. Use these if you have particularly large jobs that need lots of memory.
# The hydras all share the same data file system. This allows you to upload a file to any of the hydras and then have it on all of them.
# [[OpenMPI]] is installed
#* This allows MPI based parallelization to be used
#* Note that the hydras ARE NOT network connected, so MPI can only be used within a single hydra. It cannot communicate across multiple hydras.
#* If you need to run something in parallel across multiple nodes, you should use [[PSC]] instead.

''See [[what_resources_when| What resources when]] for guides to help decide which resource you should be using.''

== Using the servers respectfully ==
Because the servers are a shared department resource it is important to use them respectfully. There are many servers and multiple cores (CPUs) per server. To be respectful of other users:

# '''Check the current usage''' of the server you're on before using it.
# Do not use '''all''' the cores on a server, and do not use '''all''' the servers at the same time, so others can get work done.
# If you have a long-running job, '''deprioritize it''' so others can run too.

Each of these is easy to check.

=== Checking current usage ===
To check which servers are currently being used and which are open, see the [https://info.stat.cmu.edu/grafana/d/fdkb6l4klj400d/hydra-cluster?orgId=1 Grafana dashboard for the hydras]. It lists them all and their current usage.

When you're logged in to one node, you can use the <code>top</code> command. <code>top</code> displays the current usage of the server’s resources with a list of jobs run. It also tells you the percentage of the cores that are being used and the amount of RAM (memory) that is available. See [https://www.howtogeek.com/668986/how-to-use-the-linux-top-command-and-understand-its-output/ howtogeek] resources on how to interpret it all (the documentation is very clean, so we encourage you to read up ther

=== Limiting the cores you use ===
We recommend that when using parallel processing, use fewer than the total number of cores on the server. Most parallel processing packages let you choose the number of cores to use; the hydras have 12, so pick something smaller.

=== Deprioritize when possible ===
<code>nice/renice</code> allows you to lower the priority of your jobs, so the operating system lets other processes run first. See [https://www.howtogeek.com/411979/how-to-set-process-priorities-with-the-nice-and-renice-commands-in-linux/ howtogeek] resources on the basics. <code>nice</code> sets the priority, and <code>renice</code> updates your priority. On the department servers, everyone starts at the same priority, and you can only '''deprioritize''' your jobs -- you can't make them higher priority than everyone else.<syntaxhighlight lang="bash">nice -20 Rscript some-big-script.R
renice -n 20 processid # get the process ID from top</syntaxhighlight>
== Introduction to connecting to the server and useful Linux commands ==

The department servers all run Linux, and require you to use a command line to interact with it (as compared to Rstudio and other GUIs).
=== Getting Access: (<code>ssh</code>) ===

To get onto the server and type code you’ll need to access it through <code>ssh</code>.

<ul>
<li>For Macs (and Linux), open up a <code>terminal</code> and type
<syntaxhighlight lang="bash">ssh andrew_id@hydra1.stat.cmu.edu</syntaxhighlight>

you will then have to type in your password (if you don’t know it or don’t know it or have never accessed the server before - email [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu]).
</li>
<li>
For Windows, there's a few options (with different pros and cons). [https://www.makeuseof.com/tag/4-easy-ways-to-use-ssh-in-windows/ Makeuseof] provides an overview and introduction to how to get started on a few approaches, including PuTTY and Windows PowerShell.

Again, make sure to email [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu] to check that you have an account / or what your password is if you forgot.
</li>
</ul>
If you’re accessing the servers a lot, you might also look into aliases and password storing for the <code>ssh</code> commands. - Mac/Linux: [https://scotch.io/tutorials/how-to-create-an-ssh-shortcut scotch.io] - Windows: Powershell: [https://docs.microsoft.com/en-us/windows/terminal/tutorials/ssh microsoft], PuTTY: [https://www.makeuseof.com/tag/4-easy-ways-to-use-ssh-in-windows/ makeuseof]

=== Passwordless ssh ===
To get on the server without typing in the password, you need to first generate a SSH key pair.

* For Linux, open up a terminal and check first that the specific ssh key pair does not exist in the given folder.
* Next, generate a key pair (both public and private key). There are different types of keys (RSA, DSA, ECDSA, and ED25519). The <code>-t</code> option specifies the type of key pair. <code>ssh-keygen -t rsa</code>

* A prompt for what to name the key pair and what folder to place it in should appear. The username should be the username for your local computer and id_rsa is the key pair name. You can change it. The local key should be stored in the <code>~/.ssh</code> folder on your local computer. <code>Generating public/private rsa key pair.</code> <code>Enter file in which to save the key (/home/username/.ssh/id_rsa):</code>

* Hit Enter to accept the default name and location, or enter the folder and name manually.
* A prompt for a passphrase should appear. This is for securing the local key (on local computer) and is optional. <code>Enter Passphrase (empty for no passphrase):</code>

* Make sure there is a <code>~/.ssh</code> folder on the server. If not, make a .ssh folder.
* Next, upload the public key to remote server. This command will upload the public key to all the hydras and ghidorahs since they share a file system. <code>ssh-copy-id andrew_id@hydra1.stat.cmu.edu</code>

* To ensure the permissions of the <code>~/.ssh</code> directory are updated: <code>chmod 700 ~/.ssh</code> <code>chmod 600 ~/.ssh/authorized_keys</code>

You should now be able to SSH into the server without being prompted for a password.

=== File sharing (<code>scp</code> +) ===

There is also command line code to copy files from your personal computer to the server. This is done with a SCP (secure copy) tool. There are also GUIs (graphical user interfaces) that allow you to “drag and drop” files from your computer to the server (and visa-versa). Currently, a lot of students use GUIs to deal with file sharing.

==== Command line <code>scp</code>====

You can file share either by using <code>scp</code>, for example

<ol>
<li> Mac/Linux: This command is very similar to <code>cp</code> (the file copier).
<ol>
<li>Copy from server
<syntaxhighlight lang="bash"> scp username@from_host:path/to/file.txt /local/directory/ </syntaxhighlight>
<syntaxhighlight lang="bash">scp -r username@from_host:path/to/folder/ local/directory/</syntaxhighlight>
</li>
<li>Copy to server
 
<syntaxhighlight lang="bash"> scp local/file.txt username@to_host:/remote/directory/</syntaxhighlight>
<syntaxhighlight lang="bash"> scp -r local/directory/ username@to_host:/remote/directory/</syntaxhighlight>
</li>
</ol>

<code>scp</code> references: [https://haydenjames.io/linux-securely-copy-files-using-scp/ haydenjames.io]

</li>
<li>Windows: Again, this process is a bit more complicated.
<ol>
<li>A PuTTY + Command Prompt provides a version that is very similar to Mac/Linux <code>scp</code> (reference: [https://success.tanaza.com/s/article/How-to-use-SCP-command-on-Windows tanaza]), and requires minor changes in the call  (<code>pscp -scp</code> instead of <code>scp)</code>. </li>
<li>A Powershell specific command is a bit more complicated / not similar to Mac/Linux <code>scp</code> (reference: [https://www.nsoftware.com/kb/articles/psserver-scp.rst nsoftware]) </li>
</ol>
</li>
</ol>

===  GUI “drag-and-drop” file sharing ===
A '''lot''' of students use '''GUI''' interfaces to transfer files instead of <code>scp</code> command line. These tools allow you to “drag and drop” files from one location to another after the connection is set up.

Some examples include:

<ol>
<li>Mac:
<ol><li>
[https://cyberduck.io/ Cyberduck]
</li></ol></li>
<li>Linux:
<ol><li>
[https://www.openssh.com/ OpenSSH]
</li></ol></li>
<li> Windows: lots of options - [https://www.thegeekstuff.com/2011/06/windows-sftp-scp-clients/ thegeekstuff]
<ol>
<li>[https://www.putty.org/ PuTTY] </li>
<li>[https://winscp.net/eng/download.php WinSCP] </li></ol>
</li>
</ol>
=== Basic CLI (Linux command line): <code>bash</code>===

Understanding the basic Linux command line is important to be able to navigate when you are actually on the server. Below are some based commands to move around the file system, manipulate files, and view storage structure.

{| class="wikitable"
|-
! command
! translation
|-
| <code>pwd</code>
| print the current working directory
|-
| <code>cd _dir_</code>
| change directory
|-
| <code>rm _file_</code>
| remove / delete file
|-
| <code>mv _file_ _dir_</code>
| move file (can change name)
|-
| <code>cp _file_ _dir_</code>
| copy file
|-
| <code>ls _dir_</code>
| list files in directory
|}

use <code>man _command_</code> to get a listing / manual of the command.

Additional resources that capture more information on the command line and are still geared to statistical computing can be found here: [http://www.jarrodmillman.com/rcsds/standard/bash.html UCB 2015 reproducible DS lecture notes].

'''TODO: add information about <code>sed</code>, <code>chmod</code>, and a link to bash files.'''

== Text editors ==

There are multiple options to view code on the server side. They include:

*vim
*… [to be added later]

== Running jobs ==

We have included thoughts about running jobs in two places.

#The first looks at tools that allow you to keep running jobs on the server when you close the connection to the server (turn off your laptop, etc.). These tools are call [[Terminal_multiplexers|terminal multiplexers]]. 
#The second set of tools focus on how to run your [[R_on_the_server|R analysis]] or [[Python_on_the_server|Python analysis]] code on the server.

== Server specs ==

Generally speaking, our servers have multiple CPU cores, but no GPU cores. For deep learning tasks that need GPUs, consider using the [[GPU Cluster|GPU cluster]].

To examine the specs yourself, use
<syntaxhighlight lang="bash">cat /proc/meminfo
cat /proc/cpuinfo</syntaxhighlight>

to see some information.

Linux and department servers

2025-06-12T15:33:36Z

Areinhar: /* Passwordless ssh */ improve formatting and clarity

The department has a set of servers, the hydras, that can help you run large analyses and gain a better understanding of common command line structure for more high scale computing. One can also view the department servers a bit more as a “sandbox” or “playground” as there are very few constraints and it’s okay to not run as polished code as you might on the [[PSC]] or Azure.

['''Request:''' Windows users (and all users) - if you see comments that are outdated / less helpful than expected please contribute.]

== What resources exist ==

# The department provides 13 servers (named {<code>hydra1</code>, ... , <code>hydra13</code>}<code>.stat.cmu.edu</code>.
# Each hydra has 12 CPU cores, and 64 GB of RAM (per core).
## The exceptions are hydra7 and hydra8, which have 128 GB of RAM. Use these if you have particularly large jobs that need lots of memory.
# The hydras all share the same data file system. This allows you to upload a file to any of the hydras and then have it on all of them.
# [[OpenMPI]] is installed
#* This allows MPI based parallelization to be used
#* Note that the hydras ARE NOT network connected, so MPI can only be used within a single hydra. It cannot communicate across multiple hydras.
#* If you need to run something in parallel across multiple nodes, you should use [[PSC]] instead.

''See [[what_resources_when| What resources when]] for guides to help decide which resource you should be using.''

== Using the servers respectfully ==
Because the servers are a shared department resource it is important to use them respectfully. There are many servers and multiple cores (CPUs) per server. To be respectful of other users:

# '''Check the current usage''' of the server you're on before using it.
# Do not use '''all''' the cores on a server, and do not use '''all''' the servers at the same time, so others can get work done.
# If you have a long-running job, '''deprioritize it''' so others can run too.

Each of these is easy to check.

=== Checking current usage ===
To check which servers are currently being used and which are open, see the [https://info.stat.cmu.edu/grafana/d/fdkb6l4klj400d/hydra-cluster?orgId=1 Grafana dashboard for the hydras]. It lists them all and their current usage.

When you're logged in to one node, you can use the <code>top</code> command. <code>top</code> displays the current usage of the server’s resources with a list of jobs run. It also tells you the percentage of the cores that are being used and the amount of RAM (memory) that is available. See [https://www.howtogeek.com/668986/how-to-use-the-linux-top-command-and-understand-its-output/ howtogeek] resources on how to interpret it all (the documentation is very clean, so we encourage you to read up ther

=== Limiting the cores you use ===
We recommend that when using parallel processing, use fewer than the total number of cores on the server. Most parallel processing packages let you choose the number of cores to use; the hydras have 12, so pick something smaller.

=== Deprioritize when possible ===
<code>nice/renice</code> allows you to lower the priority of your jobs, so the operating system lets other processes run first. See [https://www.howtogeek.com/411979/how-to-set-process-priorities-with-the-nice-and-renice-commands-in-linux/ howtogeek] resources on the basics. <code>nice</code> sets the priority, and <code>renice</code> updates your priority. On the department servers, everyone starts at the same priority, and you can only '''deprioritize''' your jobs -- you can't make them higher priority than everyone else.<syntaxhighlight lang="bash">nice -20 Rscript some-big-script.R
renice -n 20 processid # get the process ID from top</syntaxhighlight>
== Introduction to connecting to the server and useful Linux commands ==

The department servers all run Linux, and require you to use a command line to interact with it (as compared to Rstudio and other GUIs).
=== Getting Access: (<code>ssh</code>) ===

To get onto the server and type code you’ll need to access it through <code>ssh</code>.

<ul>
<li>For Macs (and Linux), open up a <code>terminal</code> and type
<syntaxhighlight lang="bash">ssh andrew_id@hydra1.stat.cmu.edu</syntaxhighlight>

you will then have to type in your password (if you don’t know it or don’t know it or have never accessed the server before - email [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu]).
</li>
<li>
For Windows, there's a few options (with different pros and cons). [https://www.makeuseof.com/tag/4-easy-ways-to-use-ssh-in-windows/ Makeuseof] provides an overview and introduction to how to get started on a few approaches, including PuTTY and Windows PowerShell.

Again, make sure to email [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu] to check that you have an account / or what your password is if you forgot.
</li>
</ul>
If you’re accessing the servers a lot, you might also look into aliases and password storing for the <code>ssh</code> commands. - Mac/Linux: [https://scotch.io/tutorials/how-to-create-an-ssh-shortcut scotch.io] - Windows: Powershell: [https://docs.microsoft.com/en-us/windows/terminal/tutorials/ssh microsoft], PuTTY: [https://www.makeuseof.com/tag/4-easy-ways-to-use-ssh-in-windows/ makeuseof]

=== Passwordless ssh ===
To get on the server without typing in the password, you need to first generate a SSH key pair.

* For Linux, open up a terminal and check first that the specific ssh key pair does not exist in the given folder.
* Next, generate a key pair (both public and private key). There are different types of keys (RSA, DSA, ECDSA, and ED25519). The <code>-t</code> option specifies the type of key pair. <code>ssh-keygen -t rsa</code>

* A prompt for what to name the key pair and what folder to place it in should appear. The username should be the username for your local computer and id_rsa is the key pair name. You can change it. The local key should be stored in the <code>~/.ssh</code> folder on your local computer. <code>Generating public/private rsa key pair.</code> <code>Enter file in which to save the key (/home/username/.ssh/id_rsa):</code>

* Hit Enter to accept the default name and location, or enter the folder and name manually.
* A prompt for a passphrase should appear. This is for securing the local key (on local computer) and is optional. <code>Enter Passphrase (empty for no passphrase):</code>

* Make sure there is a <code>~/.ssh</code> folder on the server. If not, make a .ssh folder.
* Next, upload the public key to remote server. This command will upload the public key to all the hydras and ghidorahs since they share a file system. <code>ssh-copy-id andrew_id@hydra1.stat.cmu.edu</code>

* To ensure the permissions of the <code>~/.ssh</code> directory are updated: <code>chmod 700 ~/.ssh</code> <code>chmod 600 ~/.ssh/authorized_keys</code>

You should now be able to SSH into the server without being prompted for a password.

=== File sharing (<code>scp</code> +) ===

There is also command line code to copy files from your personal computer to the server. This is done with a SCP (secure copy) tool. There are also GUIs (graphical user interfaces) that allow you to “drag and drop” files from your computer to the server (and visa-versa). Currently, a lot of students use GUIs to deal with file sharing.

==== Command line <code>scp</code>====

You can file share either by using <code>scp</code>, for example

<ol>
<li> Mac/Linux: This command is very similar to <code>cp</code> (the file copier).
<ol>
<li>Copy from server
<syntaxhighlight lang="bash"> scp username@from_host:path/to/file.txt /local/directory/ </syntaxhighlight>
<syntaxhighlight lang="bash">scp -r username@from_host:path/to/folder/ local/directory/</syntaxhighlight>
</li>
<li>Copy to server
 
<syntaxhighlight lang="bash"> scp local/file.txt username@to_host:/remote/directory/</syntaxhighlight>
<syntaxhighlight lang="bash"> scp -r local/directory/ username@to_host:/remote/directory/</syntaxhighlight>
</li>
</ol>

<code>scp</code> references: [https://haydenjames.io/linux-securely-copy-files-using-scp/ haydenjames.io]

</li>
<li>Windows: Again, this process is a bit more complicated.
<ol>
<li>A PuTTY + Command Prompt provides a version that is very similar to Mac/Linux <code>scp</code> (reference: [https://success.tanaza.com/s/article/How-to-use-SCP-command-on-Windows tanaza]), and requires minor changes in the call  (<code>pscp -scp</code> instead of <code>scp)</code>. </li>
<li>A Powershell specific command is a bit more complicated / not similar to Mac/Linux <code>scp</code> (reference: [https://www.nsoftware.com/kb/articles/psserver-scp.rst nsoftware]) </li>
</ol>
</li>
</ol>

===  GUI “drag-and-drop” file sharing ===
A '''lot''' of students use '''GUI''' interfaces to transfer files instead of <code>scp</code> command line. These tools allow you to “drag and drop” files from one location to another after the connection is set up.

Some examples include:

<ol>
<li>Mac:
<ol><li>
[https://cyberduck.io/ Cyberduck]
</li></ol></li>
<li>Linux:
<ol><li>
[https://www.openssh.com/ OpenSSH]
</li></ol></li>
<li> Windows: lots of options - [https://www.thegeekstuff.com/2011/06/windows-sftp-scp-clients/ thegeekstuff]
<ol>
<li>[https://www.putty.org/ PuTTY] </li>
<li>[https://winscp.net/eng/download.php WinSCP] </li></ol>
</li>
</ol>
=== Basic CLI (Linux command line): <code>bash</code>===

Understanding the basic Linux command line is important to be able to navigate when you are actually on the server. Below are some based commands to move around the file system, manipulate files, and view storage structure.

{| class="wikitable"
|-
! command
! translation
|-
| <code>pwd</code>
| print the current working directory
|-
| <code>cd _dir_</code>
| change directory
|-
| <code>rm _file_</code>
| remove / delete file
|-
| <code>mv _file_ _dir_</code>
| move file (can change name)
|-
| <code>cp _file_ _dir_</code>
| copy file
|-
| <code>ls _dir_</code>
| list files in directory
|}

use <code>man _command_</code> to get a listing / manual of the command.

Additional resources that capture more information on the command line and are still geared to statistical computing can be found here: [http://www.jarrodmillman.com/rcsds/standard/bash.html UCB 2015 reproducible DS lecture notes].

'''TODO: add information about <code>sed</code>, <code>chmod</code>, and a link to bash files.'''

== Text editors ==

There are multiple options to view code on the server side. They include:

*vim
*… [to be added later]

== Running jobs ==

We have included thoughts about running jobs in two places.

#The first looks at tools that allow you to keep running jobs on the server when you close the connection to the server (turn off your laptop, etc.). These tools are call [[Terminal_multiplexers|terminal multiplexers]]. 
#The second set of tools focus on how to run your [[R_on_the_server|R analysis]] or [[Python_on_the_server|Python analysis]] code on the server.

== Server specs ==

Generally speaking, our servers have multiple CPU cores, but no GPU cores. For deep learning tasks that need GPUs, consider using the [[GPU Cluster|GPU cluster]].

To examine the specs yourself, use
<syntaxhighlight lang="bash">cat /proc/meminfo
cat /proc/cpuinfo</syntaxhighlight>

to see some information.

Main Page

2025-05-22T15:49:12Z

Areinhar: unbold

Anyone can read this wiki.  If you want to write/edit stuff in the wiki, you will need to log in with your account provided by the Statistics & Data Science Department computing group. This Wiki is intended to be a repository for instruction, information and ideas related to the many computing environments available to members and affiliates of the Statistics & Data Science Department.

== Getting to know the computing resources & getting support ==

If you're interested in knowing about what resources you have access to (and/or) what resources you have access to that might solve a problem you're having, please visit our [[What_resources_when|"what resources when" article]].

To get support that is beyond this website please attend office hours below and/or email [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu] for support (for more details about how the support ticketing works through [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu] + ServiceNow please see the [[Service_now|ServiceNow]] article).

Reset your StatDS password here - https://util.stat.cmu.edu/passwordreset/

== Department Contacts ==

• [mailto:events@stat.cmu.edu events@stat.cmu.edu]: seminars, conferences & workshops, and calendar updates

• [mailto:news@stat.cmu.edu news@stat.cmu.edu]: dept/dept member-related public news & announcements

• [mailto:lists@stat.cmu.edu lists@stat.cmu.edu]: department email lists

• [mailto:webmaster@stat.cmu.edu webmaster@stat.cmu.edu]: website updates

• [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu]: computer/tech requests

• [mailto:rooms@stat.cmu.edu rooms@stat.cmu.edu]: request space for meetings and TA office hours

• [mailto:reimbursements@stat.cmu.edu reimbursements@stat.cmu.edu]: travel and reimbursement requests

== Computing Office Hours ==

These office hour opportunities provide a dedicated time for you to come ask questions about research computing.  These include code advice, how to access certain resources, what the best resources are for your project, or to learn about new connections to help you progress in your education and research. Staff Office hour times will be sent out each week on Monday as a calendar invite; accept any of the invitations to add it to your personal calendar. PhD TA Office Hours are below, unless a different time is communicated.

#PhD TA Office Hours are held by Alex Shen in Baker 136E on Tuesdays from 3 pm to 5 pm
#Staff Office Hours are held by appointment by Carl Skipper in BH 129G. Schedule via [mailto:lists@stat.cmu.edu email].

See [[Computing_office_hour_usage|computing office hour usage]] for what office hours can be used for.

== Computing Wiki Pages ==

=== Managing servers and environments ===

==== Virtual environments ====
*[[Virtual Environment Introduction|Why use virtual environments?]]
*R's [[Renv]]
*Python's [[Python_virtual_environments|virtual environments (2 options)]]
*Old instructions for [[Conda with Python (Reference Only)|conda with Python]]

==== Containers ====
*[[Introduction to containers]]
*[[Docker, and why-you-should-use-it-for-your-research]]
*[[Tutorials|Basic tutorial]]
*[[Running Docker locally and on Hydras]]

=== Department computing resources ===

==== Printing ====
*[[Department_Printers|Department Printers]]

==== Web Hosting ====
#[[How to host your webpage under stat.cmu.edu]]
#[[Bulding your website with Hugo]]

==== Servers ====
*[[Linux_and_department_servers|Department Servers]]. This wiki introduces you to the '''hydra''' servers, basic Linux commands, and more.
*#[[Terminal_multiplexers|Terminal multiplexers (screen and tmux)]]. This wiki presents tools for use on the ''hydra'' that can allow you to run jobs even when your not on the server, and much much more!
*#[[Python on the server]]. This wiki introduces uses command line approaches to running <code>python</code> on a virtual machine, and how to use jupyter notebooks through the server as well.
*#[[R on the server]]. This wiki introduces uses command line approaches to running <code>R</code> on a virtual machine

*[[Department GPU Servers]], for small jobs needing 1 GPU.
*[[GPU Cluster]], for larger jobs that may need multiple GPUs and many CPUs.
*[[PSC]], the Pittsburgh Supercomputing Center. Free accounts and usage credits are available
*#[[ACCESS]]
*#[[Slurm]]. This wiki introduces the job management system uses at PSC.
*#[[Parallelization_on_PSC/Bridges-2]]
*#[[Access PSC compute nodes directly from VSCode]]
*Azure
*#[[Azure_getting_started|Azure: getting started]]. This wiki walks you through the steps to create a virtual machine through Azure
*#[[How to obtain a virtual machine with a GPU]]
*#[[Attach a disk to a virtual machine]]

=== Development guides ===

====[[Parallel_computing|Parallelization]]====
*[[Parallelization_in_r|R-specific parallelization]]
*[[Parallelization_in_python|Python-specific parallelization]]

==== Package development ====
*[[Developing_R_packages|R package development]]
*[[Developing_python_packages|Python package development]]
[[Terminal automation]]. This wiki introduces approaches to automate Linux/OS command line calls.
*[[Bash files]]. This wiki presents potential uses of Bash files including automated file/folder creation, parallelization and much more!
*[[Make]]. This wiki presents potential uses of <code>make</code> and Makefiles including automated workflow and dependency tracking
*[[Jumping to frequently used folders and automating environment setups]]

==== Version Control ====
See the guide provided in the site of [https://36-750.github.io/tools/version-control/ 36750 - Statistical Computing]

==== Reference management ====
#Zotero
#Mendeley

==== Integrated Development Environments (IDEs) ====
#[[VSCode|VSCode]]
#PyCharm
#RStudio
#Eclipse

==== Text Editors ====
#Vim
#Nano
#Emacs
#Atom
#Sublime

See also the guide provided in the site of [https://36-750.github.io/tools/editors/ 36750 - Statistical Computing]

==== AI Tools ====
* [[ChatGPT and other LLMs]]

*[[github_copilot|GitHub Copilot]]

==== Miscellanea ====
#[[Windows_Subsystem_for_Linux_(WSL)_and_Python|Windows Subsystem for Linux (WSL) and Python]]
#[[Amphetamine|Amphetamine: How to keep your computer awake]]

=== General Department Computing Resources ===

*Fall 2021 [[Computing_seminar_resources|Computing Seminar Resources]] (video and slides)

== Ideas for the Wiki ==

Please see the [[TODO|TODO page]] to post suggestions and errata.

== Using the Wiki ==

*Here is some [[Generic_Wiki_Information|Generic Wiki Information]] that you may be interested in, such as a users' guide and other resources for Wiki users.

Main Page

2025-05-05T18:23:25Z

Areinhar: /* Computing Wiki Pages */ add some context

Anyone can read this wiki.  If you want to write/edit stuff in the wiki, you will need to log in with your account provided by the Statistics & Data Science Department computing group. This Wiki is intended to be a repository for instruction, information and ideas related to the many computing environments available to members and affiliates of the Statistics & Data Science Department.

== Getting to know the computing resources & getting support ==

If you're interested in knowing about what resources you have access to (and/or) what resources you have access to that might solve a problem you're having, please visit our [[What_resources_when|"what resources when" article]].

To get support that is beyond this website please attend office hours below and/or email [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu] for support (for more details about how the support ticketing works through [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu] + ServiceNow please see the [[Service_now|ServiceNow]] article).

Reset your StatDS password here - https://util.stat.cmu.edu/passwordreset/

== Department Contacts ==

• [mailto:events@stat.cmu.edu events@stat.cmu.edu]: seminars, conferences & workshops, and calendar updates

• [mailto:news@stat.cmu.edu news@stat.cmu.edu]: dept/dept member-related public news & announcements

• [mailto:lists@stat.cmu.edu lists@stat.cmu.edu]: department email lists

• [mailto:webmaster@stat.cmu.edu webmaster@stat.cmu.edu]: website updates

• [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu]: computer/tech requests

• [mailto:rooms@stat.cmu.edu rooms@stat.cmu.edu]: request space for meetings and TA office hours

• [mailto:reimbursements@stat.cmu.edu reimbursements@stat.cmu.edu]: travel and reimbursement requests

== Computing Office Hours ==

These office hour opportunities provide a dedicated time for you to come ask questions about research computing.  These include code advice, how to access certain resources, what the best resources are for your project, or to learn about new connections to help you progress in your education and research. Staff Office hour times will be sent out each week on Monday as a calendar invite; accept any of the invitations to add it to your personal calendar. PhD TA Office Hours are below, unless a different time is communicated.

#PhD TA Office Hours are held by Alex Shen in Baker 136E on Tuesdays from 3 pm to 5 pm
#Staff Office Hours are held by appointment by Carl Skipper in BH 129G. Schedule via [mailto:lists@stat.cmu.edu email].

See [[Computing_office_hour_usage|computing office hour usage]] for what office hours can be used for.

== Computing Wiki Pages ==

=== Managing servers and environments ===

==== '''Virtual environments''' ====
*[[Virtual Environment Introduction|Why use virtual environments?]]
*R's [[Renv]]
*Python's [[Python_virtual_environments|virtual environments (2 options)]]
*Old instructions for [[Conda with Python (Reference Only)|conda with Python]]

==== '''Containers''' ====
*[[Introduction to containers]]
*[[Docker, and why-you-should-use-it-for-your-research]]
*[[Tutorials|Basic tutorial]]
*[[Running Docker locally and on Hydras]]

=== Department computing resources ===

==== '''Printing''' ====
*[[Department_Printers|Department Printers]]

==== '''Web Hosting''' ====
#[[How to host your webpage under stat.cmu.edu]]
#[[Bulding your website with Hugo]]

==== Servers ====
*[[Linux_and_department_servers|Department Servers]]. This wiki introduces you to the '''hydra''' servers, basic Linux commands, and more.
*#[[Terminal_multiplexers|Terminal multiplexers (screen and tmux)]]. This wiki presents tools for use on the ''hydra'' that can allow you to run jobs even when your not on the server, and much much more!
*#[[Python on the server]]. This wiki introduces uses command line approaches to running <code>python</code> on a virtual machine, and how to use jupyter notebooks through the server as well.
*#[[R on the server]]. This wiki introduces uses command line approaches to running <code>R</code> on a virtual machine

*[[Department GPU Servers]], for small jobs needing 1 GPU.
*[[GPU Cluster]], for larger jobs that may need multiple GPUs and many CPUs.
*[[PSC]], the Pittsburgh Supercomputing Center. Free accounts and usage credits are available
*#[[ACCESS]]
*#[[Slurm]]. This wiki introduces the job management system uses at PSC.
*#[[Parallelization_on_PSC/Bridges-2]]
*#[[Access PSC compute nodes directly from VSCode]]
*Azure
*#[[Azure_getting_started|Azure: getting started]]. This wiki walks you through the steps to create a virtual machine through Azure
*#[[How to obtain a virtual machine with a GPU]]
*#[[Attach a disk to a virtual machine]]

=== Development guides ===

==== '''[[Parallel_computing|Parallelization]]''' ====
*[[Parallelization_in_r|R-specific parallelization]]
*[[Parallelization_in_python|Python-specific parallelization]]

==== '''Package development''' ====
*[[Developing_R_packages|R package development]]
*[[Developing_python_packages|Python package development]]
[[Terminal automation]]. This wiki introduces approaches to automate Linux/OS command line calls.
*[[Bash files]]. This wiki presents potential uses of Bash files including automated file/folder creation, parallelization and much more!
*[[Make]]. This wiki presents potential uses of <code>make</code> and Makefiles including automated workflow and dependency tracking
*[[Jumping to frequently used folders and automating environment setups]]

==== '''Version Control''' ====
See the guide provided in the site of [https://36-750.github.io/tools/version-control/ 36750 - Statistical Computing]

==== '''Reference management''' ====
#Zotero
#Mendeley

==== '''Integrated Development Environments (IDEs)''' ====
#[[VSCode|VSCode]]
#PyCharm
#RStudio
#Eclipse

==== '''Text Editors''' ====
#Vim
#Nano
#Emacs
#Atom
#Sublime

See also the guide provided in the site of [https://36-750.github.io/tools/editors/ 36750 - Statistical Computing]

==== '''AI Tools''' ====
* [[ChatGPT and other LLMs]]

*[[github_copilot|GitHub Copilot]]

==== '''Miscellanea''' ====
#[[Windows_Subsystem_for_Linux_(WSL)_and_Python|Windows Subsystem for Linux (WSL) and Python]]
#[[Amphetamine|Amphetamine: How to keep your computer awake]]

=== General Department Computing Resources ===

*Fall 2021 [[Computing_seminar_resources|Computing Seminar Resources]] (video and slides)

== Ideas for the Wiki ==

Please see the [[TODO|TODO page]] to post suggestions and errata.

== Using the Wiki ==

*Here is some [[Generic_Wiki_Information|Generic Wiki Information]] that you may be interested in, such as a users' guide and other resources for Wiki users.

Main Page

2025-05-05T18:22:45Z

Areinhar: /* Computing Wiki Pages */ An attempt at reorganizing the list of wiki pages in a sensible way

Anyone can read this wiki.  If you want to write/edit stuff in the wiki, you will need to log in with your account provided by the Statistics & Data Science Department computing group. This Wiki is intended to be a repository for instruction, information and ideas related to the many computing environments available to members and affiliates of the Statistics & Data Science Department.

== Getting to know the computing resources & getting support ==

If you're interested in knowing about what resources you have access to (and/or) what resources you have access to that might solve a problem you're having, please visit our [[What_resources_when|"what resources when" article]].

To get support that is beyond this website please attend office hours below and/or email [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu] for support (for more details about how the support ticketing works through [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu] + ServiceNow please see the [[Service_now|ServiceNow]] article).

Reset your StatDS password here - https://util.stat.cmu.edu/passwordreset/

== Department Contacts ==

• [mailto:events@stat.cmu.edu events@stat.cmu.edu]: seminars, conferences & workshops, and calendar updates

• [mailto:news@stat.cmu.edu news@stat.cmu.edu]: dept/dept member-related public news & announcements

• [mailto:lists@stat.cmu.edu lists@stat.cmu.edu]: department email lists

• [mailto:webmaster@stat.cmu.edu webmaster@stat.cmu.edu]: website updates

• [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu]: computer/tech requests

• [mailto:rooms@stat.cmu.edu rooms@stat.cmu.edu]: request space for meetings and TA office hours

• [mailto:reimbursements@stat.cmu.edu reimbursements@stat.cmu.edu]: travel and reimbursement requests

== Computing Office Hours ==

These office hour opportunities provide a dedicated time for you to come ask questions about research computing.  These include code advice, how to access certain resources, what the best resources are for your project, or to learn about new connections to help you progress in your education and research. Staff Office hour times will be sent out each week on Monday as a calendar invite; accept any of the invitations to add it to your personal calendar. PhD TA Office Hours are below, unless a different time is communicated.

#PhD TA Office Hours are held by Alex Shen in Baker 136E on Tuesdays from 3 pm to 5 pm
#Staff Office Hours are held by appointment by Carl Skipper in BH 129G. Schedule via [mailto:lists@stat.cmu.edu email].

See [[Computing_office_hour_usage|computing office hour usage]] for what office hours can be used for.

== Computing Wiki Pages ==

=== Managing servers and environments ===

==== '''Virtual environments''' ====
*[[Virtual Environment Introduction|Why use virtual environments?]]
*R's [[Renv]]
*Python's [[Python_virtual_environments|virtual environments (2 options)]]
*Old instructions for [[Conda with Python (Reference Only)|conda with Python]]

==== '''Containers''' ====
*[[Introduction to containers]]
*[[Docker, and why-you-should-use-it-for-your-research]]
*[[Tutorials|Basic tutorial]]
*[[Running Docker locally and on Hydras]]

=== Department computing resources ===

==== '''Printing''' ====
*[[Department_Printers|Department Printers]]

==== '''Web Hosting''' ====
#[[How to host your webpage under stat.cmu.edu]]
#[[Bulding your website with Hugo]]

==== Servers ====
*[[Linux_and_department_servers|Department Servers]]. This wiki introduces you to the '''hydra''' servers, basic Linux commands, and more.
*#[[Terminal_multiplexers|Terminal multiplexers (screen and tmux)]]. This wiki presents tools for use on the ''hydra'' that can allow you to run jobs even when your not on the server, and much much more!
*#[[Python on the server]]. This wiki introduces uses command line approaches to running <code>python</code> on a virtual machine, and how to use jupyter notebooks through the server as well.
*#[[R on the server]]. This wiki introduces uses command line approaches to running <code>R</code> on a virtual machine

*[[Department GPU Servers]]
*[[GPU Cluster]]
*[[PSC]], the Pittsburgh Supercomputing Center. Free accounts and usage credits are available
*#[[ACCESS]]
*#[[Slurm]]. This wiki introduces the job management system uses at PSC.
*#[[Parallelization_on_PSC/Bridges-2]]
*#[[Access PSC compute nodes directly from VSCode]]
*Azure
*#[[Azure_getting_started|Azure: getting started]]. This wiki walks you through the steps to create a virtual machine through Azure
*#[[How to obtain a virtual machine with a GPU]]
*#[[Attach a disk to a virtual machine]]

=== Development guides ===

==== '''[[Parallel_computing|Parallelization]]''' ====
*[[Parallelization_in_r|R-specific parallelization]]
*[[Parallelization_in_python|Python-specific parallelization]]

==== '''Package development''' ====
*[[Developing_R_packages|R package development]]
*[[Developing_python_packages|Python package development]]
[[Terminal automation]]. This wiki introduces approaches to automate Linux/OS command line calls.
*[[Bash files]]. This wiki presents potential uses of Bash files including automated file/folder creation, parallelization and much more!
*[[Make]]. This wiki presents potential uses of <code>make</code> and Makefiles including automated workflow and dependency tracking
*[[Jumping to frequently used folders and automating environment setups]]

==== '''Version Control''' ====
See the guide provided in the site of [https://36-750.github.io/tools/version-control/ 36750 - Statistical Computing]

==== '''Reference management''' ====
#Zotero
#Mendeley

==== '''Integrated Development Environments (IDEs)''' ====
#[[VSCode|VSCode]]
#PyCharm
#RStudio
#Eclipse

==== '''Text Editors''' ====
#Vim
#Nano
#Emacs
#Atom
#Sublime

See also the guide provided in the site of [https://36-750.github.io/tools/editors/ 36750 - Statistical Computing]

==== '''AI Tools''' ====
* [[ChatGPT and other LLMs]]

*[[github_copilot|GitHub Copilot]]

==== '''Miscellanea''' ====
#[[Windows_Subsystem_for_Linux_(WSL)_and_Python|Windows Subsystem for Linux (WSL) and Python]]
#[[Amphetamine|Amphetamine: How to keep your computer awake]]

=== General Department Computing Resources ===

*Fall 2021 [[Computing_seminar_resources|Computing Seminar Resources]] (video and slides)

== Ideas for the Wiki ==

Please see the [[TODO|TODO page]] to post suggestions and errata.

== Using the Wiki ==

*Here is some [[Generic_Wiki_Information|Generic Wiki Information]] that you may be interested in, such as a users' guide and other resources for Wiki users.

ACCESS

2025-04-25T14:51:58Z

Areinhar: mention template letter

The NSF’s ACCESS (Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support) program builds upon the successes of the 11-year XSEDE project.

NSF funded supercomputers, like PSC are available via the ACCESS federated system.  Besides parallel or memory-intensive computing options, there are many GPU resources available to ACCESS users via the Credit Marketplace.

#If you are a Graduate Student, all you need is '''an account''',''' '''your''' CV''', a '''1-paragraph abstract''' and '''a letter from your advisor'''.  (Carl has a template letter your advisor can sign.) With these 4 items, you can complete a simple form and get up to 400,000 CPU-hours to exchange for access to most of the nation's supercomputing cyberinfrastructure. Check out the following [https://allocations.access-ci.org/preparing-your-explore-access-request link].
#If you previously held an XSEDE account, you can login here: [https://identity.access-ci.org/new-user-federated https://identity.access-ci.org/new-user-federated] and complete your account transition from XSEDE to ACCESS by selecting “ACCESS CI (XSEDE)” and subsequently login using Andrew Web Login.  You will then be guided through a short verification process similar to any other website. 
#If you are '''new''' to the department or supercomputing in general, I set up a new account so you will be able to access (no pun intended) these resources easily now or in the future.  Visit [https://identity.access-ci.org/new-user-federated https://identity.access-ci.org/new-user-federated] to set up a new account -  make sure to choose “'''Carnegie Mellon University'''”! 
#The interface to the Supercomputing Credit Market is still being improved, but we are working on documentation to address that.

This will shortly be updated to add easy access to ACCESS On-Ramps: <div id="resource-catalog-react"></div>

Wright GPU Cluster

2025-04-04T15:03:41Z

Areinhar: Mention new node and condo model

Linux and department servers

2025-01-24T16:09:58Z

Areinhar: intro

The department has a set of servers, the hydras, that can help you run large analyses and gain a better understanding of common command line structure for more high scale computing. One can also view the department servers a bit more as a “sandbox” or “playground” as there are very few constraints and it’s okay to not run as polished code as you might on the [[PSC]] or Azure.

['''Request:''' Windows users (and all users) - if you see comments that are outdated / less helpful than expected please contribute.]

== What resources exist ==

# The department provides 13 servers (named {<code>hydra1</code>, ... , <code>hydra13</code>}<code>.stat.cmu.edu</code>.
# Each hydra has 12 CPU cores, and 64 GB of RAM (per core).
## The exceptions are hydra7 and hydra8, which have 128 GB of RAM. Use these if you have particularly large jobs that need lots of memory.
# The hydras all share the same data file system. This allows you to upload a file to any of the hydras and then have it on all of them.
# [[OpenMPI]] is installed (which should allow parallelization across each server)

''See [[what_resources_when| What resources when]] for guides to help decide which resource you should be using.''

== Using the servers respectfully ==
Because the servers are a shared department resource it is important to use them respectfully. There are many servers and multiple cores (CPUs) per server. To be respectful of other users:

# '''Check the current usage''' of the server you're on before using it.
# Do not use '''all''' the cores on a server, and do not use '''all''' the servers at the same time, so others can get work done.
# If you have a long-running job, '''deprioritize it''' so others can run too.

Each of these is easy to check.

=== Checking current usage ===
To check which servers are currently being used and which are open, see the [https://info.stat.cmu.edu/grafana/d/fdkb6l4klj400d/hydra-cluster?orgId=1 Grafana dashboard for the hydras]. It lists them all and their current usage.

When you're logged in to one node, you can use the <code>top</code> command. <code>top</code> displays the current usage of the server’s resources with a list of jobs run. It also tells you the percentage of the cores that are being used and the amount of RAM (memory) that is available. See [https://www.howtogeek.com/668986/how-to-use-the-linux-top-command-and-understand-its-output/ howtogeek] resources on how to interpret it all (the documentation is very clean, so we encourage you to read up ther

=== Limiting the cores you use ===
We recommend that when using parallel processing, use fewer than the total number of cores on the server. Most parallel processing packages let you choose the number of cores to use; the hydras have 12, so pick something smaller.

=== Deprioritize when possible ===
<code>nice/renice</code> allows you to lower the priority of your jobs, so the operating system lets other processes run first. See [https://www.howtogeek.com/411979/how-to-set-process-priorities-with-the-nice-and-renice-commands-in-linux/ howtogeek] resources on the basics. <code>nice</code> sets the priority, and <code>renice</code> updates your priority. On the department servers, everyone starts at the same priority, and you can only '''deprioritize''' your jobs -- you can't make them higher priority than everyone else.<syntaxhighlight lang="bash">nice -20 Rscript some-big-script.R
renice -n 20 processid # get the process ID from top</syntaxhighlight>
== Introduction to connecting to the server and useful Linux commands ==

The department servers all run Linux, and require you to use a command line to interact with it (as compared to Rstudio and other GUIs).
=== Getting Access: (<code>ssh</code>) ===

To get onto the server and type code you’ll need to access it through <code>ssh</code>.

<ul>
<li>For Macs (and Linux), open up a <code>terminal</code> and type
<syntaxhighlight lang="bash">ssh andrew_id@hydra1.stat.cmu.edu</syntaxhighlight>

you will then have to type in your password (if you don’t know it or don’t know it or have never accessed the server before - email [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu]).
</li>
<li>
For Windows, there's a few options (with different pros and cons). [https://www.makeuseof.com/tag/4-easy-ways-to-use-ssh-in-windows/ Makeuseof] provides an overview and introduction to how to get started on a few approaches, including PuTTY and Windows PowerShell.

Again, make sure to email [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu] to check that you have an account / or what your password is if you forgot.
</li>
</ul>
If you’re accessing the servers a lot, you might also look into aliases and password storing for the <code>ssh</code> commands. - Mac/Linux: [https://scotch.io/tutorials/how-to-create-an-ssh-shortcut scotch.io] - Windows: Powershell: [https://docs.microsoft.com/en-us/windows/terminal/tutorials/ssh microsoft], PuTTY: [https://www.makeuseof.com/tag/4-easy-ways-to-use-ssh-in-windows/ makeuseof]

=== File sharing (<code>scp</code> +) ===

There is also command line code to copy files from your personal computer to the server. This is done with a SCP (secure copy) tool. There are also GUIs (graphical user interfaces) that allow you to “drag and drop” files from your computer to the server (and visa-versa). Currently, a lot of students use GUIs to deal with file sharing.

==== Command line <code>scp</code>====

You can file share either by using <code>scp</code>, for example

<ol>
<li> Mac/Linux: This command is very similar to <code>cp</code> (the file copier).
<ol>
<li>Copy from server
<syntaxhighlight lang="bash"> scp username@from_host:path/to/file.txt /local/directory/ </syntaxhighlight>
<syntaxhighlight lang="bash">scp -r username@from_host:path/to/folder/ local/directory/</syntaxhighlight>
</li>
<li>Copy to server
 
<syntaxhighlight lang="bash"> scp local/file.txt username@to_host:/remote/directory/</syntaxhighlight>
<syntaxhighlight lang="bash"> scp -r local/directory/ username@to_host:/remote/directory/</syntaxhighlight>
</li>
</ol>

<code>scp</code> references: [https://haydenjames.io/linux-securely-copy-files-using-scp/ haydenjames.io]

</li>
<li>Windows: Again, this process is a bit more complicated.
<ol>
<li>A PuTTY + Command Prompt provides a version that is very similar to Mac/Linux <code>scp</code> (reference: [https://success.tanaza.com/s/article/How-to-use-SCP-command-on-Windows tanaza]), and requires minor changes in the call  (<code>pscp -scp</code> instead of <code>scp)</code>. </li>
<li>A Powershell specific command is a bit more complicated / not similar to Mac/Linux <code>scp</code> (reference: [https://www.nsoftware.com/kb/articles/psserver-scp.rst nsoftware]) </li>
</ol>
</li>
</ol>

===  GUI “drag-and-drop” file sharing ===
A '''lot''' of students use '''GUI''' interfaces to transfer files instead of <code>scp</code> command line. These tools allow you to “drag and drop” files from one location to another after the connection is set up.

Some examples include:

<ol>
<li>Mac:
<ol><li>
[https://cyberduck.io/ Cyberduck]
</li></ol></li>
<li>Linux:
<ol><li>
[https://www.openssh.com/ OpenSSH]
</li></ol></li>
<li> Windows: lots of options - [https://www.thegeekstuff.com/2011/06/windows-sftp-scp-clients/ thegeekstuff]
<ol>
<li>[https://www.putty.org/ PuTTY] </li>
<li>[https://winscp.net/eng/download.php WinSCP] </li></ol>
</li>
</ol>
=== Basic CLI (Linux command line): <code>bash</code>===

Understanding the basic Linux command line is important to be able to navigate when you are actually on the server. Below are some based commands to move around the file system, manipulate files, and view storage structure.

{| class="wikitable"
|-
! command
! translation
|-
| <code>pwd</code>
| print the current working directory
|-
| <code>cd _dir_</code>
| change directory
|-
| <code>rm _file_</code>
| remove / delete file
|-
| <code>mv _file_ _dir_</code>
| move file (can change name)
|-
| <code>cp _file_ _dir_</code>
| copy file
|-
| <code>ls _dir_</code>
| list files in directory
|}

use <code>man _command_</code> to get a listing / manual of the command.

Additional resources that capture more information on the command line and are still geared to statistical computing can be found here: [http://www.jarrodmillman.com/rcsds/standard/bash.html UCB 2015 reproducible DS lecture notes].

'''TODO: add information about <code>sed</code>, <code>chmod</code>, and a link to bash files.'''

== Text editors ==

There are multiple options to view code on the server side. They include:

*vim
*… [to be added later]

== Running jobs ==

We have included thoughts about running jobs in two places.

#The first looks at tools that allow you to keep running jobs on the server when you close the connection to the server (turn off your laptop, etc.). These tools are call [[Terminal_multiplexers|terminal multiplexers]]. 
#The second set of tools focus on how to run your [[R_on_the_server|R analysis]] or [[Python_on_the_server|Python analysis]] code on the server.

== Server specs ==

Generally speaking, our servers have multiple CPU cores, but no GPU cores. For deep learning tasks that need GPUs, consider using the [[GPU Cluster|GPU cluster]].

To examine the specs yourself, use
<syntaxhighlight lang="bash">cat /proc/meminfo
cat /proc/cpuinfo</syntaxhighlight>

to see some information.

Linux and department servers

2025-01-24T16:00:05Z

Areinhar: /* Deprioritize when possible */ fix bolding

['''Request:''' Windows users (and all users) - if you see comments that are outdated / less helpful than expected please contribute.]

== Introduction (why use the department servers / philosophical use) ==

The department has a set of servers (hosted virtually through campus computing) that can help you process analysis faster and gain a better understanding of common command line structure for more high scale computing. One can also view the department servers a bit more as a “sandbox” or “playground” as there are very few constraints and it’s ok to not run as polished code as you might on the PSC or Azure.

== What resources exist ==

# The department provides 13 servers (named {<code>hydra1</code>, ... , <code>hydra13</code>}<code>.stat.cmu.edu</code>.
# Each hydra has 12 CPU cores, and 64 GB of RAM (per core).
## The exceptions are hydra7 and hydra8, which have 128 GB of RAM. Use these if you have particularly large jobs that need lots of memory.
# The hydras all share the same data file system. This allows you to upload a file to any of the hydras and then have it on all of them.
# [[OpenMPI]] is installed (which should allow parallelization across each server)

''See [[what_resources_when| What resources when]] for guides to help decide which resource you should be using.''

== Using the servers respectfully ==
Because the servers are a shared department resource it is important to use them respectfully. There are many servers and multiple cores (CPUs) per server. To be respectful of other users:

# '''Check the current usage''' of the server you're on before using it.
# Do not use '''all''' the cores on a server, and do not use '''all''' the servers at the same time, so others can get work done.
# If you have a long-running job, '''deprioritize it''' so others can run too.

Each of these is easy to check.

=== Checking current usage ===
To check which servers are currently being used and which are open, see the [https://info.stat.cmu.edu/grafana/d/fdkb6l4klj400d/hydra-cluster?orgId=1 Grafana dashboard for the hydras]. It lists them all and their current usage.

When you're logged in to one node, you can use the <code>top</code> command. <code>top</code> displays the current usage of the server’s resources with a list of jobs run. It also tells you the percentage of the cores that are being used and the amount of RAM (memory) that is available. See [https://www.howtogeek.com/668986/how-to-use-the-linux-top-command-and-understand-its-output/ howtogeek] resources on how to interpret it all (the documentation is very clean, so we encourage you to read up ther

=== Limiting the cores you use ===
We recommend that when using parallel processing, use fewer than the total number of cores on the server. Most parallel processing packages let you choose the number of cores to use; the hydras have 12, so pick something smaller.

=== Deprioritize when possible ===
<code>nice/renice</code> allows you to lower the priority of your jobs, so the operating system lets other processes run first. See [https://www.howtogeek.com/411979/how-to-set-process-priorities-with-the-nice-and-renice-commands-in-linux/ howtogeek] resources on the basics. <code>nice</code> sets the priority, and <code>renice</code> updates your priority. On the department servers, everyone starts at the same priority, and you can only '''deprioritize''' your jobs -- you can't make them higher priority than everyone else.<syntaxhighlight lang="bash">nice -20 Rscript some-big-script.R
renice -n 20 processid # get the process ID from top</syntaxhighlight>
== Introduction to connecting to the server and useful Linux commands ==

The department servers all run Linux, and require you to use a command line to interact with it (as compared to Rstudio and other GUIs).
=== Getting Access: (<code>ssh</code>) ===

To get onto the server and type code you’ll need to access it through <code>ssh</code>.

<ul>
<li>For Macs (and Linux), open up a <code>terminal</code> and type
<syntaxhighlight lang="bash">ssh andrew_id@hydra1.stat.cmu.edu</syntaxhighlight>

you will then have to type in your password (if you don’t know it or don’t know it or have never accessed the server before - email [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu]).
</li>
<li>
For Windows, there's a few options (with different pros and cons). [https://www.makeuseof.com/tag/4-easy-ways-to-use-ssh-in-windows/ Makeuseof] provides an overview and introduction to how to get started on a few approaches, including PuTTY and Windows PowerShell.

Again, make sure to email [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu] to check that you have an account / or what your password is if you forgot.
</li>
</ul>
If you’re accessing the servers a lot, you might also look into aliases and password storing for the <code>ssh</code> commands. - Mac/Linux: [https://scotch.io/tutorials/how-to-create-an-ssh-shortcut scotch.io] - Windows: Powershell: [https://docs.microsoft.com/en-us/windows/terminal/tutorials/ssh microsoft], PuTTY: [https://www.makeuseof.com/tag/4-easy-ways-to-use-ssh-in-windows/ makeuseof]

=== File sharing (<code>scp</code> +) ===

There is also command line code to copy files from your personal computer to the server. This is done with a SCP (secure copy) tool. There are also GUIs (graphical user interfaces) that allow you to “drag and drop” files from your computer to the server (and visa-versa). Currently, a lot of students use GUIs to deal with file sharing.

==== Command line <code>scp</code>====

You can file share either by using <code>scp</code>, for example

<ol>
<li> Mac/Linux: This command is very similar to <code>cp</code> (the file copier).
<ol>
<li>Copy from server
<syntaxhighlight lang="bash"> scp username@from_host:path/to/file.txt /local/directory/ </syntaxhighlight>
<syntaxhighlight lang="bash">scp -r username@from_host:path/to/folder/ local/directory/</syntaxhighlight>
</li>
<li>Copy to server
 
<syntaxhighlight lang="bash"> scp local/file.txt username@to_host:/remote/directory/</syntaxhighlight>
<syntaxhighlight lang="bash"> scp -r local/directory/ username@to_host:/remote/directory/</syntaxhighlight>
</li>
</ol>

<code>scp</code> references: [https://haydenjames.io/linux-securely-copy-files-using-scp/ haydenjames.io]

</li>
<li>Windows: Again, this process is a bit more complicated.
<ol>
<li>A PuTTY + Command Prompt provides a version that is very similar to Mac/Linux <code>scp</code> (reference: [https://success.tanaza.com/s/article/How-to-use-SCP-command-on-Windows tanaza]), and requires minor changes in the call  (<code>pscp -scp</code> instead of <code>scp)</code>. </li>
<li>A Powershell specific command is a bit more complicated / not similar to Mac/Linux <code>scp</code> (reference: [https://www.nsoftware.com/kb/articles/psserver-scp.rst nsoftware]) </li>
</ol>
</li>
</ol>

===  GUI “drag-and-drop” file sharing ===
A '''lot''' of students use '''GUI''' interfaces to transfer files instead of <code>scp</code> command line. These tools allow you to “drag and drop” files from one location to another after the connection is set up.

Some examples include:

<ol>
<li>Mac:
<ol><li>
[https://cyberduck.io/ Cyberduck]
</li></ol></li>
<li>Linux:
<ol><li>
[https://www.openssh.com/ OpenSSH]
</li></ol></li>
<li> Windows: lots of options - [https://www.thegeekstuff.com/2011/06/windows-sftp-scp-clients/ thegeekstuff]
<ol>
<li>[https://www.putty.org/ PuTTY] </li>
<li>[https://winscp.net/eng/download.php WinSCP] </li></ol>
</li>
</ol>
=== Basic CLI (Linux command line): <code>bash</code>===

Understanding the basic Linux command line is important to be able to navigate when you are actually on the server. Below are some based commands to move around the file system, manipulate files, and view storage structure.

{| class="wikitable"
|-
! command
! translation
|-
| <code>pwd</code>
| print the current working directory
|-
| <code>cd _dir_</code>
| change directory
|-
| <code>rm _file_</code>
| remove / delete file
|-
| <code>mv _file_ _dir_</code>
| move file (can change name)
|-
| <code>cp _file_ _dir_</code>
| copy file
|-
| <code>ls _dir_</code>
| list files in directory
|}

use <code>man _command_</code> to get a listing / manual of the command.

Additional resources that capture more information on the command line and are still geared to statistical computing can be found here: [http://www.jarrodmillman.com/rcsds/standard/bash.html UCB 2015 reproducible DS lecture notes].

'''TODO: add information about <code>sed</code>, <code>chmod</code>, and a link to bash files.'''

== Text editors ==

There are multiple options to view code on the server side. They include:

*vim
*… [to be added later]

== Running jobs ==

We have included thoughts about running jobs in two places.

#The first looks at tools that allow you to keep running jobs on the server when you close the connection to the server (turn off your laptop, etc.). These tools are call [[Terminal_multiplexers|terminal multiplexers]]. 
#The second set of tools focus on how to run your [[R_on_the_server|R analysis]] or [[Python_on_the_server|Python analysis]] code on the server.

== Server specs ==

Generally speaking, our servers have multiple CPU cores, but no GPU cores. For deep learning tasks that need GPUs, consider using the [[GPU Cluster|GPU cluster]].

To examine the specs yourself, use
<syntaxhighlight lang="bash">cat /proc/meminfo
cat /proc/cpuinfo</syntaxhighlight>

to see some information.

Linux and department servers

2025-01-24T15:59:24Z

Areinhar: /* Using the server respectfully */ Expand section and reorganize page

['''Request:''' Windows users (and all users) - if you see comments that are outdated / less helpful than expected please contribute.]

== Introduction (why use the department servers / philosophical use) ==

The department has a set of servers (hosted virtually through campus computing) that can help you process analysis faster and gain a better understanding of common command line structure for more high scale computing. One can also view the department servers a bit more as a “sandbox” or “playground” as there are very few constraints and it’s ok to not run as polished code as you might on the PSC or Azure.

== What resources exist ==

# The department provides 13 servers (named {<code>hydra1</code>, ... , <code>hydra13</code>}<code>.stat.cmu.edu</code>.
# Each hydra has 12 CPU cores, and 64 GB of RAM (per core).
## The exceptions are hydra7 and hydra8, which have 128 GB of RAM. Use these if you have particularly large jobs that need lots of memory.
# The hydras all share the same data file system. This allows you to upload a file to any of the hydras and then have it on all of them.
# [[OpenMPI]] is installed (which should allow parallelization across each server)

''See [[what_resources_when| What resources when]] for guides to help decide which resource you should be using.''

== Using the servers respectfully ==
Because the servers are a shared department resource it is important to use them respectfully. There are many servers and multiple cores (CPUs) per server. To be respectful of other users:

# '''Check the current usage''' of the server you're on before using it.
# Do not use '''all''' the cores on a server, and do not use '''all''' the servers at the same time, so others can get work done.
# If you have a long-running job, '''deprioritize it''' so others can run too.

Each of these is easy to check.

=== Checking current usage ===
To check which servers are currently being used and which are open, see the [https://info.stat.cmu.edu/grafana/d/fdkb6l4klj400d/hydra-cluster?orgId=1 Grafana dashboard for the hydras]. It lists them all and their current usage.

When you're logged in to one node, you can use the <code>top</code> command. <code>top</code> displays the current usage of the server’s resources with a list of jobs run. It also tells you the percentage of the cores that are being used and the amount of RAM (memory) that is available. See [https://www.howtogeek.com/668986/how-to-use-the-linux-top-command-and-understand-its-output/ howtogeek] resources on how to interpret it all (the documentation is very clean, so we encourage you to read up ther

=== Limiting the cores you use ===
We recommend that when using parallel processing, use fewer than the total number of cores on the server. Most parallel processing packages let you choose the number of cores to use; the hydras have 12, so pick something smaller.

=== '''Deprioritize when possible''' ===
<code>nice/renice</code> allows you to lower the priority of your jobs, so the operating system lets other processes run first. See [https://www.howtogeek.com/411979/how-to-set-process-priorities-with-the-nice-and-renice-commands-in-linux/ howtogeek] resources on the basics. <code>nice</code> sets the priority, and <code>renice</code> updates your priority. On the department servers, everyone starts at the same priority, and you can only '''deprioritize''' your jobs -- you can't make them higher priority than everyone else.<syntaxhighlight lang="bash">nice -20 Rscript some-big-script.R
renice -n 20 processid # get the process ID from top</syntaxhighlight>
== Introduction to connecting to the server and useful Linux commands ==

The department servers all run Linux, and require you to use a command line to interact with it (as compared to Rstudio and other GUIs).
=== Getting Access: (<code>ssh</code>) ===

To get onto the server and type code you’ll need to access it through <code>ssh</code>.

<ul>
<li>For Macs (and Linux), open up a <code>terminal</code> and type
<syntaxhighlight lang="bash">ssh andrew_id@hydra1.stat.cmu.edu</syntaxhighlight>

you will then have to type in your password (if you don’t know it or don’t know it or have never accessed the server before - email [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu]).
</li>
<li>
For Windows, there's a few options (with different pros and cons). [https://www.makeuseof.com/tag/4-easy-ways-to-use-ssh-in-windows/ Makeuseof] provides an overview and introduction to how to get started on a few approaches, including PuTTY and Windows PowerShell.

Again, make sure to email [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu] to check that you have an account / or what your password is if you forgot.
</li>
</ul>
If you’re accessing the servers a lot, you might also look into aliases and password storing for the <code>ssh</code> commands. - Mac/Linux: [https://scotch.io/tutorials/how-to-create-an-ssh-shortcut scotch.io] - Windows: Powershell: [https://docs.microsoft.com/en-us/windows/terminal/tutorials/ssh microsoft], PuTTY: [https://www.makeuseof.com/tag/4-easy-ways-to-use-ssh-in-windows/ makeuseof]

=== File sharing (<code>scp</code> +) ===

There is also command line code to copy files from your personal computer to the server. This is done with a SCP (secure copy) tool. There are also GUIs (graphical user interfaces) that allow you to “drag and drop” files from your computer to the server (and visa-versa). Currently, a lot of students use GUIs to deal with file sharing.

==== Command line <code>scp</code>====

You can file share either by using <code>scp</code>, for example

<ol>
<li> Mac/Linux: This command is very similar to <code>cp</code> (the file copier).
<ol>
<li>Copy from server
<syntaxhighlight lang="bash"> scp username@from_host:path/to/file.txt /local/directory/ </syntaxhighlight>
<syntaxhighlight lang="bash">scp -r username@from_host:path/to/folder/ local/directory/</syntaxhighlight>
</li>
<li>Copy to server
 
<syntaxhighlight lang="bash"> scp local/file.txt username@to_host:/remote/directory/</syntaxhighlight>
<syntaxhighlight lang="bash"> scp -r local/directory/ username@to_host:/remote/directory/</syntaxhighlight>
</li>
</ol>

<code>scp</code> references: [https://haydenjames.io/linux-securely-copy-files-using-scp/ haydenjames.io]

</li>
<li>Windows: Again, this process is a bit more complicated.
<ol>
<li>A PuTTY + Command Prompt provides a version that is very similar to Mac/Linux <code>scp</code> (reference: [https://success.tanaza.com/s/article/How-to-use-SCP-command-on-Windows tanaza]), and requires minor changes in the call  (<code>pscp -scp</code> instead of <code>scp)</code>. </li>
<li>A Powershell specific command is a bit more complicated / not similar to Mac/Linux <code>scp</code> (reference: [https://www.nsoftware.com/kb/articles/psserver-scp.rst nsoftware]) </li>
</ol>
</li>
</ol>

===  GUI “drag-and-drop” file sharing ===
A '''lot''' of students use '''GUI''' interfaces to transfer files instead of <code>scp</code> command line. These tools allow you to “drag and drop” files from one location to another after the connection is set up.

Some examples include:

<ol>
<li>Mac:
<ol><li>
[https://cyberduck.io/ Cyberduck]
</li></ol></li>
<li>Linux:
<ol><li>
[https://www.openssh.com/ OpenSSH]
</li></ol></li>
<li> Windows: lots of options - [https://www.thegeekstuff.com/2011/06/windows-sftp-scp-clients/ thegeekstuff]
<ol>
<li>[https://www.putty.org/ PuTTY] </li>
<li>[https://winscp.net/eng/download.php WinSCP] </li></ol>
</li>
</ol>
=== Basic CLI (Linux command line): <code>bash</code>===

Understanding the basic Linux command line is important to be able to navigate when you are actually on the server. Below are some based commands to move around the file system, manipulate files, and view storage structure.

{| class="wikitable"
|-
! command
! translation
|-
| <code>pwd</code>
| print the current working directory
|-
| <code>cd _dir_</code>
| change directory
|-
| <code>rm _file_</code>
| remove / delete file
|-
| <code>mv _file_ _dir_</code>
| move file (can change name)
|-
| <code>cp _file_ _dir_</code>
| copy file
|-
| <code>ls _dir_</code>
| list files in directory
|}

use <code>man _command_</code> to get a listing / manual of the command.

Additional resources that capture more information on the command line and are still geared to statistical computing can be found here: [http://www.jarrodmillman.com/rcsds/standard/bash.html UCB 2015 reproducible DS lecture notes].

'''TODO: add information about <code>sed</code>, <code>chmod</code>, and a link to bash files.'''

== Text editors ==

There are multiple options to view code on the server side. They include:

*vim
*… [to be added later]

== Running jobs ==

We have included thoughts about running jobs in two places.

#The first looks at tools that allow you to keep running jobs on the server when you close the connection to the server (turn off your laptop, etc.). These tools are call [[Terminal_multiplexers|terminal multiplexers]]. 
#The second set of tools focus on how to run your [[R_on_the_server|R analysis]] or [[Python_on_the_server|Python analysis]] code on the server.

== Server specs ==

Generally speaking, our servers have multiple CPU cores, but no GPU cores. For deep learning tasks that need GPUs, consider using the [[GPU Cluster|GPU cluster]].

To examine the specs yourself, use
<syntaxhighlight lang="bash">cat /proc/meminfo
cat /proc/cpuinfo</syntaxhighlight>

to see some information.

What resources when

2025-01-24T15:32:08Z

Areinhar: /* The hydras (the department servers) */ Correct size info on hydras

To try to answer the common question “what resources do I have access to and when should I use them”, below we present to approaches to answering the question. The first of which focuses on what resources exist and what they should be used for and the second tries to answer resource needs / current computing problems with resources to fit those needs/ solve those problems.

After you’re found the resource you think you’d like to use (or even when you’re still deciding), we strongly encourage you to attend an computing office hour to help discuss all the things you should be thinking about and get you on the right track quickly.

= What computing resources do you have access to =

== Your personal computer ==

Your personal computer is great for small jobs that take less than 30 minutes, uses few cores and not too much RAM.

'''Pro:'''

#Naturally the easiest tool to use since it’s yours. Best for short jobs that require only a few cores.

'''Cons:'''

#
capacity:

#*few cores: 4-8 cores common on laptops<ref><p>From [http://laptopmag.com/articles/Cpu-comparison laptopmag]</p></ref> (multiply processes per cors means = 8-16 “cores”).
#*low RAM: 4-32 GB of common on laptops<ref><p>From [http://digitaltrends.com/computing/how-much-ram digitaltrends]</p></ref>, used across all cores
#*“no” GPUs: you don’t have GPUs that are optimized for Deep Learning

#
other usage: You often want to use your computer for other things (and it’s built for that exact thing). It’s possible you or someone you know has has had to leave their laptop at home for the day/ run lode oernight When they need to run something intensive -OR- long. But you need that laptop for other work / homework…

The below XKCD comic doesn’t have to be you… (though breaks are good):

[[File:Compiling.png|thumb|none|https://xkcd.com/303/]]

'''Conclusion:''' Your personal computer isn’t the best for when doing a lot of computing and its easy to put jobs elsewhere!

'''Common use-cases'''

#Most class projects
#Non-data problems
#Non-complex structure problems

'''Associated Articles''':

*[[Parallel_computing|parallelization]]

== The <code>hydras</code> (the department servers) ==

The departmental servers should be seen as a “sandbox”. You can experiment with lots of things on the servers and it doesn’t cost you much to do. The hydras have a decent amount of RAM (much more than your personal computer), but often it won’t run a lot faster than if you put you’re computer aside and did the same thing on it (though a large pro is you don’t have to stop using your own computer when running jobs on the hydras).

'''Pros:'''

#Best for long but smaller core jobs
#Decent amount of RAM per machine (64 GB, or 128 GB on hydra7 and hydra8)
#Very low barrier to entry (aka easy to get started)
#File management system shares across different hydra servers

'''Cons:'''

#usage: The department server is a bit of a “wild-west” in terms of usage (it varies how many people use it at a given time)
#cores per server: the number of cores per server is only 12

'''Conclusion:''' Great place to start for more intensive computing.

'''Associated Articles''':

*[[Linux_and_department_servers|Linux and department servers]]
'''Server Monitoring'''

View current Hydra resource usage - [https://info.stat.cmu.edu/grafana/d/fdkb6l4klj400d/hydra-cluster?orgId=1 Hydra Cluster Usage Monitor]

== PSC (Pittsburgh supercomputer center) ==

The Pittsburgh supercomputer center provides supercomputers than have massive amounts of cores (but also can find some with large RAMs and GPUs). Generally speaking, the supercomputing resources are often used when you have extremely parallelizable code that would take a very long time if you ran it on just a few cores (e.g. your computer or a few hydras).

'''Pros:'''

#A massive amount of cores and RAM that can be leveraged for extremely parallelizable computing
#Easier to access in the statistics department

'''Cons:'''

#Requires learning of new job management system (though quick to learn - see [Slurm])
#Some options to access the resources can take time or require going through your advisor
#''currently (2021)'' has been having problems with servers crashing

'''Associated Articles''':

*[[PSC|PSC]]
*[[Slurm|Slurm]]

== Azure ==

Getting an virtual Azure machine is often only needed if you need access to GPUs and you want complete prioritization of your jobs. There are other reasons to utilize Azure, but there are start-up costs and there are only so many department credits to go around (that doesn’t mean you shouldn’t use it).

'''Pros:'''

#personal virtual computing that can be used only by you (maybe also a small lab group)
#very flexibly it what is available (e.g. #cores, RAM, GPU vs CPU, etc)

'''Cons:'''

#Start-up costs are higher (requires more understanding of system administration)
#Requires more clear understanding of the costs as it isn’t free
#Some resources might not be available relative to department allocations

'''Associated Articles''':

*[[Azure_getting_started|Azure: getting started]]

== GPU Computing ==
The Statistics & Data Science Department offers two main GPU computing environments.

'''Ghidorahs'''

* 3 physical servers
* RTX 4090 GPUs
* Connected to the Stats network share
* Your statistics directory account will grant you access

'''Wright'''

* For a detail overview, please check [[GPU Cluster|here]]
* You must request access
* The Statistics network share is not accessible

= Computing resource flowchart =

These flow charts try to help capture what’s available and when you might use them.

Visit the links below to view current computing resource usage. Select a server with the least amount of users and lowest resource usage.

* [https://info.stat.cmu.edu/grafana/d/fdkb6l4klj400d/hydra-cluster?orgId=1 Hydra Cluster Usage Monitor]

* [https://info.stat.cmu.edu/grafana/d/edkar1rgy3l6od/ghidorah-cluster?orgId=1 Ghidorah Cluster Usage Monitor]
== [[File:ProblemsOnYourPersonalComputer11.png|706x706px]]==

From [http://laptopmag.com/articles/Cpu-comparison laptopmag]
<br/>      2. <font color="blue">↑</font>
From [http://digitaltrends.com/computing/how-much-ram digitaltrends]

Linux and department servers

2025-01-24T15:30:38Z

Areinhar: /* What resources exist */ Mention nodes with extra RAM

['''Request:''' Windows users (and all users) - if you see comments that are outdated / less helpful than expected please contribute.]

== Introduction (why use the department servers / philosophical use) ==

The department has a set of servers (hosted virtually through campus computing) that can help you process analysis faster and gain a better understanding of common command line structure for more high scale computing. One can also view the department servers a bit more as a “sandbox” or “playground” as there are very few constraints and it’s ok to not run as polished code as you might on the PSC or Azure.

== What resources exist ==

# The department provides 13 servers (named {<code>hydra1</code>, ... , <code>hydra13</code>}<code>.stat.cmu.edu</code>.
# Each hydra has 12 CPU cores, and 64 GB of RAM (per core).
## The exceptions are hydra7 and hydra8, which have 128 GB of RAM. Use these if you have particularly large jobs that need lots of memory.
# The hydras all share the same data file system. This allows you to upload a file to any of the hydras and then have it on all of them.
# [[OpenMPI]] is installed (which should allow parallelization across each server)

''See [[what_resources_when| What resources when]] for guides to help decide which resource you should be using.''

== Introduction to connecting to the server and useful Linux commands ==

The department servers all run Linux, and require you to use a command line to interact with it (as compared to Rstudio and other GUIs).
=== Getting Access: (<code>ssh</code>) ===

To get onto the server and type code you’ll need to access it through <code>ssh</code>.

<ul>
<li>For Macs (and Linux), open up a <code>terminal</code> and type
<syntaxhighlight lang="bash">ssh andrew_id@hydra1.stat.cmu.edu</syntaxhighlight>

you will then have to type in your password (if you don’t know it or don’t know it or have never accessed the server before - email [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu]).
</li>
<li>
For Windows, there's a few options (with different pros and cons). [https://www.makeuseof.com/tag/4-easy-ways-to-use-ssh-in-windows/ Makeuseof] provides an overview and introduction to how to get started on a few approaches, including PuTTY and Windows PowerShell.

Again, make sure to email [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu] to check that you have an account / or what your password is if you forgot.
</li>
</ul>
If you’re accessing the servers a lot, you might also look into aliases and password storing for the <code>ssh</code> commands. - Mac/Linux: [https://scotch.io/tutorials/how-to-create-an-ssh-shortcut scotch.io] - Windows: Powershell: [https://docs.microsoft.com/en-us/windows/terminal/tutorials/ssh microsoft], PuTTY: [https://www.makeuseof.com/tag/4-easy-ways-to-use-ssh-in-windows/ makeuseof]

=== File sharing (<code>scp</code> +) ===

There is also command line code to copy files from your personal computer to the server. This is done with a SCP (secure copy) tool. There are also GUIs (graphical user interfaces) that allow you to “drag and drop” files from your computer to the server (and visa-versa). Currently, a lot of students use GUIs to deal with file sharing.

==== Command line <code>scp</code>====

You can file share either by using <code>scp</code>, for example

<ol>
<li> Mac/Linux: This command is very similar to <code>cp</code> (the file copier).
<ol>
<li>Copy from server
 
<syntaxhighlight lang="bash"> scp username@from_host:path/to/file.txt /local/directory/ </syntaxhighlight>
<syntaxhighlight lang="bash">scp -r username@from_host:path/to/folder/ local/directory/</syntaxhighlight>
</li>
<li>Copy to server
 
<syntaxhighlight lang="bash"> scp local/file.txt username@to_host:/remote/directory/</syntaxhighlight>
<syntaxhighlight lang="bash"> scp -r local/directory/ username@to_host:/remote/directory/</syntaxhighlight>
</li>
</ol>

<code>scp</code> references: [https://haydenjames.io/linux-securely-copy-files-using-scp/ haydenjames.io]

</li>
<li>Windows: Again, this process is a bit more complicated.
<ol>
<li>A PuTTY + Command Prompt provides a version that is very similar to Mac/Linux <code>scp</code> (reference: [https://success.tanaza.com/s/article/How-to-use-SCP-command-on-Windows tanaza]), and requires minor changes in the call  (<code>pscp -scp</code> instead of <code>scp)</code>. </li>
<li>A Powershell specific command is a bit more complicated / not similar to Mac/Linux <code>scp</code> (reference: [https://www.nsoftware.com/kb/articles/psserver-scp.rst nsoftware]) </li>
</ol>
</li>
</ol>

 

==== Command line <code>sftp</code>====

[to do: add section on sftp]

==== GUI “drag and drop” file sharing ====

A '''lot''' of students use '''GUI''' interfaces to transfer files instead of <code>scp</code> command line. These tools allow you to “drag and drop” files from one location to another after the connection is set up.

Some examples include:

<ol>
<li>Mac:
<ol><li>
[https://cyberduck.io/ Cyberduck]
</li></ol></li>
<li>Linux:
<ol><li>
[https://www.openssh.com/ OpenSSH]
</li></ol></li>
<li> Windows: lots of options - [https://www.thegeekstuff.com/2011/06/windows-sftp-scp-clients/ thegeekstuff]
<ol>
<li>[https://www.putty.org/ PuTTY] </li>
<li>[https://winscp.net/eng/download.php WinSCP] </li>
<li>… </li>
</ol>
</li>
</ol>

=== Basic CLI (Linux command line): <code>bash</code>===

Understanding the basic Linux command line is important to be able to navigate when you are actually on the server. Below are some based commands to move around the file system, manipulate files, and view storage structure.

{| class="wikitable"
|-
! command
! translation
|-
| <code>pwd</code>
| present working directory
|-
| <code>cd _dir_</code>
| change directory
|-
| <code>rm _file_</code>
| remove / delete file
|-
| <code>mv _file_ _dir_</code>
| move file (can change name)
|-
| <code>cp _file_ _dir_</code>
| copy file
|-
| <code>ls _dir_</code>
| list files in directory
|}

use <code>man _command_</code> to get a listing / manual of the command.

Additional resources that capture more information on the command line and are still geared to statistical computing can be found here: [http://www.jarrodmillman.com/rcsds/standard/bash.html UCB 2015 reproducible DS lecture notes].

'''TODO: add information about <code>sed</code>, <code>chmod</code>, and a link to bash files.'''

== Text editors ==

There are multiple options to view code on the server side. They include:

*vim
*… [to be added later]

== Running jobs ==

We have included thoughts about running jobs in two places.

#The first looks at tools that allow you to keep running jobs on the server when you close the connection to the server (turn off your laptop, etc.). These tools are call [[Terminal_multiplexers|terminal multiplexers]]. 
#The second set of tools focus on how to run your [[R_on_the_server|R analysis]] or [[Python_on_the_server|Python analysis]] code on the server.

== Server specs ==

Generally speaking, our servers have multiple CPU cores, but no GPU cores. For deep learning tasks that need GPUs, consider using the [[GPU Cluster|GPU cluster]].

To examine the specs yourself, use
<syntaxhighlight lang="bash">cat /proc/meminfo
cat /proc/cpuinfo</syntaxhighlight>

to see some information.
== Using the server respectfully ==

Because the servers are a shared department resource it is important to use them respectfully. There are many servers and multiple cores (CPUs) per server. To be respectful of other users we recommend checking (1) the current usage of any server you’re on (it’s possible one server is being used a lot but others are not), (2) not run a parallelize process that could use '''all''' of a servers cores (at least leave 1 core open), and (3) deprioritize your jobs that you don’t need to have done soon (especially if others are using the server when you submit your job).

<ol>
<li>'''Understanding current resource usage:'''<br/> <code>top</code> lets you know the current usage of the server’s resources with a list of jobs run. It also tells you the percentage of the cores that are being used and the amount of RAM (memory) that is being used/ what is available. See [https://www.howtogeek.com/668986/how-to-use-the-linux-top-command-and-understand-its-output/ howtogeek] resources on how to interpret it all (the documentation is very clean, so we encourage you to read up there).
<syntaxhighlight lang="bash">top</syntaxhighlight>
</li>

<li>'''Parallelizing with less than 100% of the cores:'''<br/>To avoid using all the cores when parallelizing, we recommend you first look at <code>top</code>, and then at least do <code>(# of total cores) -1</code> as the upper limit of the number of cores used (less if others are using it as well). </li>
<li>'''Deprioritizing when possible:'''<br/> <code>nice/renice</code> allows you to lower the priority of your jobs (so that others may use the resources if necessary) See [https://www.howtogeek.com/411979/how-to-set-process-priorities-with-the-nice-and-renice-commands-in-linux/ howtogeek] resources on the basics. <code>nice</code> sets the priority, and <code>renice</code> updates your priority. On the department servers, your can only '''deprioritize''' your jobs, everyone starts out on the same level.
<syntaxhighlight lang="bash">nice --20 ./andrew_id
renice</syntaxhighlight>
</li>
</ol>

Linux and department servers

2025-01-10T16:11:43Z

Areinhar: fix /proc/ typos, clean up headings

['''Request:''' Windows users (and all users) - if you see comments that are outdated / less helpful than expected please contribute.]

== Introduction (why use the department servers / philosophical use) ==

The department has a set of servers (hosted virtually through campus computing) that can help you process analysis faster and gain a better understanding of common command line structure for more high scale computing. One can also view the department servers a bit more as a “sandbox” or “playground” as there are very few constraints and it’s ok to not run as polished code as you might on the PSC or Azure.

== What resources exist ==

# The department provides 13 servers (named {<code>hydra1</code>, ... , <code>hydra13</code>}<code>.stat.cmu.edu</code>.
# Each hydra has 12 CPU cores, and 64 GB of RAM (per core).
# The hydras all share the same data file system. This allows you to upload a file to any of the hydras and then have it on all of them.
# [[OpenMPI]] is installed (which should allow parallelization across each server)

''See [[what_resources_when| What resources when]] for guides to help decide which resource you should be using.''

== Introduction to connecting to the server and useful Linux commands ==

The department servers all run Linux, and require you to use a command line to interact with it (as compared to Rstudio and other GUIs).
=== Getting Access: (<code>ssh</code>) ===

To get onto the server and type code you’ll need to access it through <code>ssh</code>.

<ul>
<li>For Macs (and Linux), open up a <code>terminal</code> and type
<syntaxhighlight lang="bash">ssh andrew_id@hydra1.stat.cmu.edu</syntaxhighlight>

you will then have to type in your password (if you don’t know it or don’t know it or have never accessed the server before - email [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu]).
</li>
<li>
For Windows, there's a few options (with different pros and cons). [https://www.makeuseof.com/tag/4-easy-ways-to-use-ssh-in-windows/ Makeuseof] provides an overview and introduction to how to get started on a few approaches, including PuTTY and Windows PowerShell.

Again, make sure to email [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu] to check that you have an account / or what your password is if you forgot.
</li>
</ul>
If you’re accessing the servers a lot, you might also look into aliases and password storing for the <code>ssh</code> commands. - Mac/Linux: [https://scotch.io/tutorials/how-to-create-an-ssh-shortcut scotch.io] - Windows: Powershell: [https://docs.microsoft.com/en-us/windows/terminal/tutorials/ssh microsoft], PuTTY: [https://www.makeuseof.com/tag/4-easy-ways-to-use-ssh-in-windows/ makeuseof]

=== File sharing (<code>scp</code> +) ===

There is also command line code to copy files from your personal computer to the server. This is done with a SCP (secure copy) tool. There are also GUIs (graphical user interfaces) that allow you to “drag and drop” files from your computer to the server (and visa-versa). Currently, a lot of students use GUIs to deal with file sharing.

==== Command line <code>scp</code>====

You can file share either by using <code>scp</code>, for example

<ol>
<li> Mac/Linux: This command is very similar to <code>cp</code> (the file copier).
<ol>
<li>Copy from server
 
<syntaxhighlight lang="bash"> scp username@from_host:path/to/file.txt /local/directory/ </syntaxhighlight>
<syntaxhighlight lang="bash">scp -r username@from_host:path/to/folder/ local/directory/</syntaxhighlight>
</li>
<li>Copy to server
 
<syntaxhighlight lang="bash"> scp local/file.txt username@to_host:/remote/directory/</syntaxhighlight>
<syntaxhighlight lang="bash"> scp -r local/directory/ username@to_host:/remote/directory/</syntaxhighlight>
</li>
</ol>

<code>scp</code> references: [https://haydenjames.io/linux-securely-copy-files-using-scp/ haydenjames.io]

</li>
<li>Windows: Again, this process is a bit more complicated.
<ol>
<li>A PuTTY + Command Prompt provides a version that is very similar to Mac/Linux <code>scp</code> (reference: [https://success.tanaza.com/s/article/How-to-use-SCP-command-on-Windows tanaza]), and requires minor changes in the call  (<code>pscp -scp</code> instead of <code>scp)</code>. </li>
<li>A Powershell specific command is a bit more complicated / not similar to Mac/Linux <code>scp</code> (reference: [https://www.nsoftware.com/kb/articles/psserver-scp.rst nsoftware]) </li>
</ol>
</li>
</ol>

 

==== Command line <code>sftp</code>====

[to do: add section on sftp]

==== GUI “drag and drop” file sharing ====

A '''lot''' of students use '''GUI''' interfaces to transfer files instead of <code>scp</code> command line. These tools allow you to “drag and drop” files from one location to another after the connection is set up.

Some examples include:

<ol>
<li>Mac:
<ol><li>
[https://cyberduck.io/ Cyberduck]
</li></ol></li>
<li>Linux:
<ol><li>
[https://www.openssh.com/ OpenSSH]
</li></ol></li>
<li> Windows: lots of options - [https://www.thegeekstuff.com/2011/06/windows-sftp-scp-clients/ thegeekstuff]
<ol>
<li>[https://www.putty.org/ PuTTY] </li>
<li>[https://winscp.net/eng/download.php WinSCP] </li>
<li>… </li>
</ol>
</li>
</ol>

=== Basic CLI (Linux command line): <code>bash</code>===

Understanding the basic Linux command line is important to be able to navigate when you are actually on the server. Below are some based commands to move around the file system, manipulate files, and view storage structure.

{| class="wikitable"
|-
! command
! translation
|-
| <code>pwd</code>
| present working directory
|-
| <code>cd _dir_</code>
| change directory
|-
| <code>rm _file_</code>
| remove / delete file
|-
| <code>mv _file_ _dir_</code>
| move file (can change name)
|-
| <code>cp _file_ _dir_</code>
| copy file
|-
| <code>ls _dir_</code>
| list files in directory
|}

use <code>man _command_</code> to get a listing / manual of the command.

Additional resources that capture more information on the command line and are still geared to statistical computing can be found here: [http://www.jarrodmillman.com/rcsds/standard/bash.html UCB 2015 reproducible DS lecture notes].

'''TODO: add information about <code>sed</code>, <code>chmod</code>, and a link to bash files.'''

== Text editors ==

There are multiple options to view code on the server side. They include:

*vim
*… [to be added later]

== Running jobs ==

We have included thoughts about running jobs in two places.

#The first looks at tools that allow you to keep running jobs on the server when you close the connection to the server (turn off your laptop, etc.). These tools are call [[Terminal_multiplexers|terminal multiplexers]]. 
#The second set of tools focus on how to run your [[R_on_the_server|R analysis]] or [[Python_on_the_server|Python analysis]] code on the server.

== Server specs ==

Generally speaking, our servers have multiple CPU cores, but no GPU cores. For deep learning tasks that need GPUs, consider using the [[GPU Cluster|GPU cluster]].

To examine the specs yourself, use
<syntaxhighlight lang="bash">cat /proc/meminfo
cat /proc/cpuinfo</syntaxhighlight>

to see some information.
== Using the server respectfully ==

Because the servers are a shared department resource it is important to use them respectfully. There are many servers and multiple cores (CPUs) per server. To be respectful of other users we recommend checking (1) the current usage of any server you’re on (it’s possible one server is being used a lot but others are not), (2) not run a parallelize process that could use '''all''' of a servers cores (at least leave 1 core open), and (3) deprioritize your jobs that you don’t need to have done soon (especially if others are using the server when you submit your job).

<ol>
<li>'''Understanding current resource usage:'''<br/> <code>top</code> lets you know the current usage of the server’s resources with a list of jobs run. It also tells you the percentage of the cores that are being used and the amount of RAM (memory) that is being used/ what is available. See [https://www.howtogeek.com/668986/how-to-use-the-linux-top-command-and-understand-its-output/ howtogeek] resources on how to interpret it all (the documentation is very clean, so we encourage you to read up there).
<syntaxhighlight lang="bash">top</syntaxhighlight>
</li>

<li>'''Parallelizing with less than 100% of the cores:'''<br/>To avoid using all the cores when parallelizing, we recommend you first look at <code>top</code>, and then at least do <code>(# of total cores) -1</code> as the upper limit of the number of cores used (less if others are using it as well). </li>
<li>'''Deprioritizing when possible:'''<br/> <code>nice/renice</code> allows you to lower the priority of your jobs (so that others may use the resources if necessary) See [https://www.howtogeek.com/411979/how-to-set-process-priorities-with-the-nice-and-renice-commands-in-linux/ howtogeek] resources on the basics. <code>nice</code> sets the priority, and <code>renice</code> updates your priority. On the department servers, your can only '''deprioritize''' your jobs, everyone starts out on the same level.
<syntaxhighlight lang="bash">nice --20 ./andrew_id
renice</syntaxhighlight>
</li>
</ol>