Building Workflows on Remote Servers: Difference between revisions
No edit summary |
No edit summary |
||
| Line 6: | Line 6: | ||
== Department Servers == | == Department Servers == | ||
The department wholly owns and operates a small HPC cluster, composed of several CPU-only nodes known as the Leviathans, and a few GPU nodes known as the Ghidorahs. The department cluster is suitable for modest sized workflows that would be too slow to run on a personal device. '''If your workflow requires continual, simultaneous use of multiple Leviathans or Ghidorahs, we recommend the other options in the next | The department wholly owns and operates a small HPC cluster, composed of several CPU-only nodes known as the Leviathans, and a few GPU nodes known as the Ghidorahs. The department cluster is suitable for modest sized workflows that would be too slow to run on a personal device. '''If your workflow requires continual, simultaneous use of multiple Leviathans or Ghidorahs, we recommend the other options in the next section.''' | ||
{| class="wikitable" | {| class="wikitable" | ||
|+ | |+ | ||
| Line 16: | Line 16: | ||
|[[Linux and department servers]] | |[[Linux and department servers]] | ||
|Introduces basic Linux commands and instructions for connecting to department servers | |Introduces basic Linux commands and instructions for connecting to department servers | ||
| | |⭐⭐⚫⚫⚫ | ||
|style="background-color: #44ee88;" | '''All server workflows''' | |style="background-color: #44ee88;" | '''All server workflows''' | ||
|- | |- | ||
| Line 28: | Line 28: | ||
|⭐⭐⚫⚫⚫ | |⭐⭐⚫⚫⚫ | ||
|Python workflows | |Python workflows | ||
|} | |||
== External Shared Servers == | |||
Students in the department also have access to other HPC resources that are shared between departments and institutions | |||
{| class="wikitable" | |||
|+ | |||
!Title | |||
!Description | |||
!Technical Level | |||
!Relevance | |||
|- | |||
|[[GPU Cluster|Wright GPU Cluster]] | |||
|Manual for accessing and using the Wright cluster | |||
|⭐⭐⭐⚫⚫ | |||
|Large-scale, GPU-heavy server workflows | |||
|- | |||
|[[PSC]] (Incomplete) | |||
|Instruction for accessing and using the Pittsburgh Supercomputing Center servers | |||
| | |||
|Large-scale server workflows | |||
|- | |||
| | |||
| | |||
| | |||
| | |||
|} | |} | ||
== Other Useful Tools == | == Other Useful Tools == | ||
Below are some useful tools and tricks for doing work on any remote server. | |||
{| class="wikitable" | {| class="wikitable" | ||
|+ | |+ | ||
| Line 38: | Line 64: | ||
!Relevance | !Relevance | ||
|- | |- | ||
|VS Code Remote ( | |[[Slurm]] | ||
|Introduces the job scheduling system using on larger HPC clusters (like Wright and PSC) | |||
| | |||
| | |||
|- | |||
|VS Code Remote (TBW) | |||
|Introduces the VS Code Remote plugin that enables the full VS Code IDE experience on a remote server | |Introduces the VS Code Remote plugin that enables the full VS Code IDE experience on a remote server | ||
| | | | ||
Revision as of 06:27, 27 March 2026
Even with perfect parallelization, research experiments can quickly scale beyond the capabilities of any personal computing device. Thus, most workflows with high computing requirements are often performed on High-Performance Computing (HPC) clusters, which are dedicated servers for large computational workloads. These servers often have hardware orders of magnitude faster than personal devices.
Students in the CMU Department of Statistics and Data Science have access to several HPC clusters. Each cluster is configured for various types and sizes of workflows.
Back to Beginner's Guide to Computation in Statistical Research
Department Servers
The department wholly owns and operates a small HPC cluster, composed of several CPU-only nodes known as the Leviathans, and a few GPU nodes known as the Ghidorahs. The department cluster is suitable for modest sized workflows that would be too slow to run on a personal device. If your workflow requires continual, simultaneous use of multiple Leviathans or Ghidorahs, we recommend the other options in the next section.
| Title | Description | Technical Level | Relevance |
|---|---|---|---|
| Linux and department servers | Introduces basic Linux commands and instructions for connecting to department servers | ⭐⭐⚫⚫⚫ | All server workflows |
| R on the server | Demonstrates how to run R scripts and RStudio on department servers | ⭐⭐⚫⚫⚫ | R worflows |
| Python on the server | Demonstrates how to run Python scripts and Jupyter notebooks on department servers | ⭐⭐⚫⚫⚫ | Python workflows |
Students in the department also have access to other HPC resources that are shared between departments and institutions
| Title | Description | Technical Level | Relevance |
|---|---|---|---|
| Wright GPU Cluster | Manual for accessing and using the Wright cluster | ⭐⭐⭐⚫⚫ | Large-scale, GPU-heavy server workflows |
| PSC (Incomplete) | Instruction for accessing and using the Pittsburgh Supercomputing Center servers | Large-scale server workflows | |
Other Useful Tools
Below are some useful tools and tricks for doing work on any remote server.
| Title | Description | Technical Level | Relevance |
|---|---|---|---|
| Slurm | Introduces the job scheduling system using on larger HPC clusters (like Wright and PSC) | ||
| VS Code Remote (TBW) | Introduces the VS Code Remote plugin that enables the full VS Code IDE experience on a remote server | ||
