Building Workflows on Remote Servers: Difference between revisions
mNo edit summary |
mNo edit summary |
||
| (7 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
[[Category:Beginner's_Guide_Scaling|01]] | |||
{{ScalingSidebar}} | |||
Even with perfect parallelization, research experiments can quickly scale beyond the capabilities of any personal computing device. Thus, most workflows with high computing requirements are often performed on High-Performance Computing (HPC) clusters, which are dedicated servers for large computational workloads. These servers often have hardware orders of magnitude faster than personal devices. | Even with perfect parallelization, research experiments can quickly scale beyond the capabilities of any personal computing device. Thus, most workflows with high computing requirements are often performed on High-Performance Computing (HPC) clusters, which are dedicated servers for large computational workloads. These servers often have hardware orders of magnitude faster than personal devices. | ||
Students in the CMU Department of Statistics and Data Science have access to several HPC clusters. Each cluster is configured for various types and sizes of workflows. | Students in the CMU Department of Statistics and Data Science have access to several HPC clusters. Each cluster is configured for various types and sizes of workflows. | ||
== Department Servers == | Back to [[Beginner's Guide to Computation in Statistical Research]] | ||
The department wholly owns and operates a small HPC cluster, composed of several CPU-only nodes known as the Leviathans, and a few GPU nodes known as the Ghidorahs. The department cluster is suitable for modest sized workflows that would be too slow to run on a personal device. '''If your workflow requires continual, simultaneous use of multiple Leviathans or Ghidorahs, we recommend the | |||
== 1. Department Servers == | |||
The department wholly owns and operates a small HPC cluster, composed of several CPU-only nodes known as the Leviathans, and a few GPU nodes known as the Ghidorahs. The department cluster is suitable for modest sized workflows that would be too slow to run on a personal device. | |||
<div style="display: grid; grid-template-columns: repeat(auto-fit, minmax(500px, 1fr))"> | |||
<div style="background-color: #fff3cd; border-left: 5px solid #ffc107; padding: 15px; margin-bottom: 20px; margin-top: 15px; overflow: hidden;"> | |||
'''Important: Resource Limits'''<br> | |||
If your workflow requires continual, simultaneous use of multiple Leviathans or Ghidorahs, we strongly recommend migrating to the external shared servers in the next section to ensure fair use for all students. | |||
</div> | |||
<div style="background-color: #f8f9fa; border-left: 4px solid #6c757d; padding: 15px; margin-bottom: 10px;"> | |||
'''[[Linux and department servers]]'''<br> | |||
Introduces basic Linux commands and instructions for connecting to the department servers. Start here for all server-based workflows. | |||
</div> | |||
<div style="background-color: #f8f9fa; border-left: 4px solid #75AADB; padding: 15px; margin-bottom: 10px;"> | |||
'''[[R on the server]]'''<br> | |||
Demonstrates how to run R scripts and RStudio on the department servers. | |||
</div> | |||
| | <div style="background-color: #f8f9fa; border-left: 4px solid #005A9C; padding: 15px; margin-bottom: 10px;"> | ||
'''[[Python on the server]]'''<br> | |||
Demonstrates how to run Python scripts and Jupyter notebooks on the department servers. | |||
</div> | |||
</div> | |||
== 2. External Shared Servers == | |||
Students in the department also have access to other HPC resources that are shared between departments and institutions. These clusters are built for massive parallelization and GPU-intensive workloads. | |||
<div style="display: grid; grid-template-columns: repeat(auto-fit, minmax(500px, 1fr))"> | |||
<div style="background-color: #f8f9fa; border-left: 4px solid #dc3545; padding: 15px; margin-bottom: 10px;"> | |||
'''[[Slurm]]'''<br> | |||
Introduces the job scheduling system used on larger HPC clusters (like Wright and PSC). Essential reading for medium to large-scale server workflows. | |||
</div> | |||
<div style="background-color: #f8f9fa; border-left: 4px solid #28a745; padding: 15px; margin-bottom: 10px;"> | |||
'''[[GPU Cluster | Wright GPU Cluster]]'''<br> | |||
The manual for accessing and using the Wright cluster. Ideal for large-scale, GPU-heavy server workflows. | |||
</div> | |||
<div style="background-color: #f8f9fa; border-left: 4px solid #6f42c1; padding: 15px; margin-bottom: 10px;"> | |||
'''[[PSC]] (Incomplete)'''<br> | |||
Instructions for accessing and using the Pittsburgh Supercomputing Center (PSC) servers for massive-scale computing. | |||
</div> | |||
</div> | |||
== 3. Other Useful Tools == | |||
Below are some useful tools and tricks for streamlining your work on any remote server. | |||
= | <div style="background-color: #f8f9fa; border-left: 4px solid #17a2b8; padding: 15px; margin-bottom: 10px;"> | ||
'''[[VS Code Remote]]'''<br> | |||
Introduces the VS Code Remote plugin, which allows you to edit files and run terminals using the full VS Code IDE experience directly on a remote server. | |||
</div> | |||
Latest revision as of 14:25, 7 July 2026
| Scaling Checklist |
|---|
| Back to Beginner's Guide |
Even with perfect parallelization, research experiments can quickly scale beyond the capabilities of any personal computing device. Thus, most workflows with high computing requirements are often performed on High-Performance Computing (HPC) clusters, which are dedicated servers for large computational workloads. These servers often have hardware orders of magnitude faster than personal devices.
Students in the CMU Department of Statistics and Data Science have access to several HPC clusters. Each cluster is configured for various types and sizes of workflows.
Back to Beginner's Guide to Computation in Statistical Research
1. Department Servers
The department wholly owns and operates a small HPC cluster, composed of several CPU-only nodes known as the Leviathans, and a few GPU nodes known as the Ghidorahs. The department cluster is suitable for modest sized workflows that would be too slow to run on a personal device.
Linux and department servers
Introduces basic Linux commands and instructions for connecting to the department servers. Start here for all server-based workflows.
R on the server
Demonstrates how to run R scripts and RStudio on the department servers.
Python on the server
Demonstrates how to run Python scripts and Jupyter notebooks on the department servers.
Students in the department also have access to other HPC resources that are shared between departments and institutions. These clusters are built for massive parallelization and GPU-intensive workloads.
Slurm
Introduces the job scheduling system used on larger HPC clusters (like Wright and PSC). Essential reading for medium to large-scale server workflows.
Wright GPU Cluster
The manual for accessing and using the Wright cluster. Ideal for large-scale, GPU-heavy server workflows.
PSC (Incomplete)
Instructions for accessing and using the Pittsburgh Supercomputing Center (PSC) servers for massive-scale computing.
3. Other Useful Tools
Below are some useful tools and tricks for streamlining your work on any remote server.
VS Code Remote
Introduces the VS Code Remote plugin, which allows you to edit files and run terminals using the full VS Code IDE experience directly on a remote server.
