Building Workflows on Remote Servers

Even with perfect parallelization, research experiments can quickly scale beyond the capabilities of any personal computing device. Thus, most workflows with high computing requirements are often performed on High-Performance Computing (HPC) clusters, which are dedicated servers for large computational workloads. These servers often have hardware orders of magnitude faster than personal devices.

Students in the CMU Department of Statistics and Data Science have access to several HPC clusters. Each cluster is configured for various types and sizes of workflows.

Back to Beginner's Guide to Computation in Statistical Research

Department Servers

The department wholly owns and operates a small HPC cluster, composed of several CPU-only nodes known as the Leviathans, and a few GPU nodes known as the Ghidorahs. The department cluster is suitable for modest sized workflows that would be too slow to run on a personal device. If your workflow requires continual, simultaneous use of multiple Leviathans or Ghidorahs, we recommend the other options in the next sections.

Title	Description	Technical Level	Relevance
Linux and department servers	Introduces basic Linux commands and instructions for connecting to department servers	⭐⚫⚫⚫⚫	All server workflows
R on the server	Demonstrates how to run R scripts and RStudio on department servers	⭐⭐⚫⚫⚫	R worflows
Python on the server	Demonstrates how to run Python scripts and Jupyter notebooks on department servers	⭐⭐⚫⚫⚫	Python workflows