Building Workflows on Remote Servers - Revision history

Areinhar at 14:25, 7 July 2026

2026-07-07T14:25:41Z

← Older revision		Revision as of 14:25, 7 July 2026
Line 1:		Line 1:
	[[Category:Beginner's_Guide_Scaling\|01]]		[[Category:Beginner's_Guide_Scaling\|01]]

	{{ScalingSidebar}}		{{ScalingSidebar}}

Areinhar: use a grid of boxes instead of a sequence

2026-07-07T14:19:51Z

use a grid of boxes instead of a sequence

← Older revision		Revision as of 14:19, 7 July 2026
Line 12:		Line 12:
	The department wholly owns and operates a small HPC cluster, composed of several CPU-only nodes known as the Leviathans, and a few GPU nodes known as the Ghidorahs. The department cluster is suitable for modest sized workflows that would be too slow to run on a personal device.		The department wholly owns and operates a small HPC cluster, composed of several CPU-only nodes known as the Leviathans, and a few GPU nodes known as the Ghidorahs. The department cluster is suitable for modest sized workflows that would be too slow to run on a personal device.

			<div style="display: grid; grid-template-columns: repeat(auto-fit, minmax(500px, 1fr))">
	<div style="background-color: #fff3cd; border-left: 5px solid #ffc107; padding: 15px; margin-bottom: 20px; margin-top: 15px; overflow: hidden;">		<div style="background-color: #fff3cd; border-left: 5px solid #ffc107; padding: 15px; margin-bottom: 20px; margin-top: 15px; overflow: hidden;">
	'''Important: Resource Limits'''<br>		'''Important: Resource Limits'''<br>
Line 30:		Line 31:
	'''[[Python on the server]]'''<br>		'''[[Python on the server]]'''<br>
	Demonstrates how to run Python scripts and Jupyter notebooks on the department servers.		Demonstrates how to run Python scripts and Jupyter notebooks on the department servers.
			</div>
	</div>		</div>

Line 35:		Line 37:
	Students in the department also have access to other HPC resources that are shared between departments and institutions. These clusters are built for massive parallelization and GPU-intensive workloads.		Students in the department also have access to other HPC resources that are shared between departments and institutions. These clusters are built for massive parallelization and GPU-intensive workloads.

			<div style="display: grid; grid-template-columns: repeat(auto-fit, minmax(500px, 1fr))">
	<div style="background-color: #f8f9fa; border-left: 4px solid #dc3545; padding: 15px; margin-bottom: 10px;">		<div style="background-color: #f8f9fa; border-left: 4px solid #dc3545; padding: 15px; margin-bottom: 10px;">
	'''[[Slurm]]'''<br>		'''[[Slurm]]'''<br>
Line 48:		Line 51:
	'''[[PSC]] (Incomplete)'''<br>		'''[[PSC]] (Incomplete)'''<br>
	Instructions for accessing and using the Pittsburgh Supercomputing Center (PSC) servers for massive-scale computing.		Instructions for accessing and using the Pittsburgh Supercomputing Center (PSC) servers for massive-scale computing.
			</div>
	</div>		</div>

Ajshen: /* 3. Other Useful Tools */

2026-06-27T05:33:54Z

3. Other Useful Tools

← Older revision		Revision as of 05:33, 27 June 2026
Line 54:		Line 54:

	<div style="background-color: #f8f9fa; border-left: 4px solid #17a2b8; padding: 15px; margin-bottom: 10px;">		<div style="background-color: #f8f9fa; border-left: 4px solid #17a2b8; padding: 15px; margin-bottom: 10px;">
	'''[[VS Code Remote]] ~~(TBW)~~'''<br>		'''[[VS Code Remote]]'''<br>
	Introduces the VS Code Remote plugin, which allows you to edit files and run terminals using the full VS Code IDE experience directly on a remote server.		Introduces the VS Code Remote plugin, which allows you to edit files and run terminals using the full VS Code IDE experience directly on a remote server.
	</div>		</div>

Ajshen at 04:40, 26 June 2026

2026-06-26T04:40:03Z

← Older revision		Revision as of 04:40, 26 June 2026
Line 1:		Line 1:
			[[Category:Beginner's_Guide_Scaling\|01]]

			{{ScalingSidebar}}

	Even with perfect parallelization, research experiments can quickly scale beyond the capabilities of any personal computing device. Thus, most workflows with high computing requirements are often performed on High-Performance Computing (HPC) clusters, which are dedicated servers for large computational workloads. These servers often have hardware orders of magnitude faster than personal devices.		Even with perfect parallelization, research experiments can quickly scale beyond the capabilities of any personal computing device. Thus, most workflows with high computing requirements are often performed on High-Performance Computing (HPC) clusters, which are dedicated servers for large computational workloads. These servers often have hardware orders of magnitude faster than personal devices.

Line 5:		Line 9:
	Back to [[Beginner's Guide to Computation in Statistical Research]]		Back to [[Beginner's Guide to Computation in Statistical Research]]

	== Department Servers ==		== 1. Department Servers ==
	The department wholly owns and operates a small HPC cluster, composed of several CPU-only nodes known as the Leviathans, and a few GPU nodes known as the Ghidorahs. The department cluster is suitable for modest sized workflows that would be too slow to run on a personal device. '''If your workflow requires continual, simultaneous use of multiple Leviathans or Ghidorahs, we recommend the ~~other options~~ in the next section.~~'''~~		The department wholly owns and operates a small HPC cluster, composed of several CPU-only nodes known as the Leviathans, and a few GPU nodes known as the Ghidorahs. The department cluster is suitable for modest sized workflows that would be too slow to run on a personal device.
	~~{\| class~~="~~wikitable~~"
	\|+		<div style="background-color: #fff3cd; border-left: 5px solid #ffc107; padding: 15px; margin-bottom: 20px; margin-top: 15px; overflow: hidden;">
	~~!Title~~		'''Important: Resource Limits'''<br>
	~~!Description~~		If your workflow requires continual, simultaneous use of multiple Leviathans or Ghidorahs, we strongly recommend migrating to the external shared servers in the next section to ensure fair use for all students.
	~~!Technical Level~~		</div>
	~~!Relevance~~
	\|-		<div style="background-color: #f8f9fa; border-left: 4px solid #6c757d; padding: 15px; margin-bottom: 10px;">
	\|[[Linux and department servers]]		'''[[Linux and department servers]]'''<br>
	\|Introduces basic Linux commands and instructions for connecting to department servers		Introduces basic Linux commands and instructions for connecting to the department servers. Start here for all server-based workflows.
	~~\|⭐⭐⚫⚫⚫~~		</div>
	\|style="background-color: #~~44ee88~~;" ~~\| '''All server workflows~~'''
	\|-		<div style="background-color: #f8f9fa; border-left: 4px solid #75AADB; padding: 15px; margin-bottom: 10px;">
	\|[[R on the server]]		'''[[R on the server]]'''<br>
	\|Demonstrates how to run R scripts and RStudio on department servers		Demonstrates how to run R scripts and RStudio on the department servers.
	~~\|⭐⭐⚫⚫⚫~~		</div>
	~~\|R worflows~~
	\|-		<div style="background-color: #f8f9fa; border-left: 4px solid #005A9C; padding: 15px; margin-bottom: 10px;">
	\|[[Python on the server]]		'''[[Python on the server]]'''<br>
	\|Demonstrates how to run Python scripts and Jupyter notebooks on department servers		Demonstrates how to run Python scripts and Jupyter notebooks on the department servers.
	\|~~⭐⭐⚫⚫⚫~~		</div>
	~~\|Python~~ workflows
	\|}		== 2. External Shared Servers ==
			Students in the department also have access to other HPC resources that are shared between departments and institutions. These clusters are built for massive parallelization and GPU-intensive workloads.

			<div style="background-color: #f8f9fa; border-left: 4px solid #dc3545; padding: 15px; margin-bottom: 10px;">
			'''[[Slurm]]'''<br>
			Introduces the job scheduling system used on larger HPC clusters (like Wright and PSC). Essential reading for medium to large-scale server workflows.
			</div>

			<div style="background-color: #f8f9fa; border-left: 4px solid #28a745; padding: 15px; margin-bottom: 10px;">
			'''[[GPU Cluster \| Wright GPU Cluster]]'''<br>
			The manual for accessing and using the Wright cluster. Ideal for large-scale, GPU-heavy server workflows.
			</div>

			<div style="background-color: #f8f9fa; border-left: 4px solid #6f42c1; padding: 15px; margin-bottom: 10px;">
			'''[[PSC]] (Incomplete)'''<br>
			Instructions for accessing and using the Pittsburgh Supercomputing Center (PSC) servers for massive-scale computing.
			</div>

	== ~~External Shared Servers~~ ==		== 3. Other Useful Tools ==
	~~Students in the department also have access to other HPC resources that~~ are ~~shared between departments~~ and ~~institutions~~		Below are some useful tools and tricks for streamlining your work on any remote server.
	~~{\| class="wikitable"~~
	\|+
	~~!Title~~
	~~!Description~~
	~~!Technical Level~~
	~~!Relevance~~
	\|-
	~~\|[[Slurm]]~~
	~~\|Introduces the job scheduling system using~~ on ~~larger HPC clusters (like Wright and PSC)~~
	~~\|⭐⭐⭐⚫⚫~~
	~~\|Medium/Large-scale server workflows~~
	\|-
	~~\|[[GPU Cluster\|Wright GPU Cluster]]~~
	~~\|Manual for accessing and using the Wright cluster~~
	~~\|⭐⭐⭐⚫⚫~~
	~~\|Large-scale, GPU-heavy server workflows~~
	\|-
	~~\|[[PSC]] (Incomplete)~~
	~~\|Instruction for accessing and using the Pittsburgh Supercomputing Center servers~~
	\|
	~~\|Large-scale~~ server ~~workflows~~
	\|-
	\|
	\|
	\|
	\|
	\|}

	~~== Other Useful Tools ==~~		<div style="background-color: #f8f9fa; border-left: 4px solid #17a2b8; padding: 15px; margin-bottom: 10px;">
	~~Below are some useful tools and tricks for doing work on any remote server.~~		'''[[VS Code Remote]] (TBW)'''<br>
	~~{\| class~~="~~wikitable~~"		Introduces the VS Code Remote plugin, which allows you to edit files and run terminals using the full VS Code IDE experience directly on a remote server.
	\|+		</div>
	~~!Title~~
	~~!Description~~
	~~!Technical Level~~
	~~!Relevance~~
	\|-
	\|VS Code Remote (TBW)
	\|Introduces the VS Code Remote plugin ~~that enables~~ the full VS Code IDE experience on a remote server
	\|
	\|
	\|-
	\|
	\|
	\|
	\|
	\|-
	\|
	\|
	\|
	\|
	\|}

Ajshen at 07:05, 10 April 2026

2026-04-10T07:05:41Z

← Older revision		Revision as of 07:05, 10 April 2026
Line 38:		Line 38:
	!Technical Level		!Technical Level
	!Relevance		!Relevance
			\|-
			\|[[Slurm]]
			\|Introduces the job scheduling system using on larger HPC clusters (like Wright and PSC)
			\|⭐⭐⭐⚫⚫
			\|Medium/Large-scale server workflows
	\|-		\|-
	\|[[GPU Cluster\|Wright GPU Cluster]]		\|[[GPU Cluster\|Wright GPU Cluster]]
Line 63:		Line 68:
	!Technical Level		!Technical Level
	!Relevance		!Relevance
	\|-
	~~\|[[Slurm]]~~
	~~\|Introduces the job scheduling system using on larger HPC clusters (like Wright and PSC)~~
	\|
	\|
	\|-		\|-
	\|VS Code Remote (TBW)		\|VS Code Remote (TBW)

Ajshen at 06:27, 27 March 2026

2026-03-27T06:27:30Z

← Older revision		Revision as of 06:27, 27 March 2026
Line 6:		Line 6:

	== Department Servers ==		== Department Servers ==
	The department wholly owns and operates a small HPC cluster, composed of several CPU-only nodes known as the Leviathans, and a few GPU nodes known as the Ghidorahs. The department cluster is suitable for modest sized workflows that would be too slow to run on a personal device. '''If your workflow requires continual, simultaneous use of multiple Leviathans or Ghidorahs, we recommend the other options in the next ~~sections~~.'''		The department wholly owns and operates a small HPC cluster, composed of several CPU-only nodes known as the Leviathans, and a few GPU nodes known as the Ghidorahs. The department cluster is suitable for modest sized workflows that would be too slow to run on a personal device. '''If your workflow requires continual, simultaneous use of multiple Leviathans or Ghidorahs, we recommend the other options in the next section.'''
	{\| class="wikitable"		{\| class="wikitable"
	\|+		\|+
Line 16:		Line 16:
	\|[[Linux and department servers]]		\|[[Linux and department servers]]
	\|Introduces basic Linux commands and instructions for connecting to department servers		\|Introduces basic Linux commands and instructions for connecting to department servers
	\|~~⭐⚫⚫⚫⚫~~		\|⭐⭐⚫⚫⚫
	\|style="background-color: #44ee88;" \| '''All server workflows'''		\|style="background-color: #44ee88;" \| '''All server workflows'''
	\|-		\|-
Line 28:		Line 28:
	\|⭐⭐⚫⚫⚫		\|⭐⭐⚫⚫⚫
	\|Python workflows		\|Python workflows
			\|}

			== External Shared Servers ==
			Students in the department also have access to other HPC resources that are shared between departments and institutions
			{\| class="wikitable"
			\|+
			!Title
			!Description
			!Technical Level
			!Relevance
			\|-
			\|[[GPU Cluster\|Wright GPU Cluster]]
			\|Manual for accessing and using the Wright cluster
			\|⭐⭐⭐⚫⚫
			\|Large-scale, GPU-heavy server workflows
			\|-
			\|[[PSC]] (Incomplete)
			\|Instruction for accessing and using the Pittsburgh Supercomputing Center servers
			\|
			\|Large-scale server workflows
			\|-
			\|
			\|
			\|
			\|
	\|}		\|}

	== Other Useful Tools ==		== Other Useful Tools ==
			Below are some useful tools and tricks for doing work on any remote server.
	{\| class="wikitable"		{\| class="wikitable"
	\|+		\|+
Line 38:		Line 64:
	!Relevance		!Relevance
	\|-		\|-
	\|VS Code Remote (~~TBD~~)		\|[[Slurm]]
			\|Introduces the job scheduling system using on larger HPC clusters (like Wright and PSC)
			\|
			\|
			\|-
			\|VS Code Remote (TBW)
	\|Introduces the VS Code Remote plugin that enables the full VS Code IDE experience on a remote server		\|Introduces the VS Code Remote plugin that enables the full VS Code IDE experience on a remote server
	\|		\|

Ajshen at 06:17, 27 March 2026

2026-03-27T06:17:13Z

← Older revision		Revision as of 06:17, 27 March 2026
Line 31:		Line 31:

	== Other Useful Tools ==		== Other Useful Tools ==
			{\| class="wikitable"
			\|+
			!Title
			!Description
			!Technical Level
			!Relevance
			\|-
			\|VS Code Remote (TBD)
			\|Introduces the VS Code Remote plugin that enables the full VS Code IDE experience on a remote server
			\|
			\|
			\|-
			\|
			\|
			\|
			\|
			\|-
			\|
			\|
			\|
			\|
			\|}

Ajshen at 06:15, 27 March 2026

2026-03-27T06:15:14Z

← Older revision		Revision as of 06:15, 27 March 2026
Line 2:		Line 2:

	Students in the CMU Department of Statistics and Data Science have access to several HPC clusters. Each cluster is configured for various types and sizes of workflows.		Students in the CMU Department of Statistics and Data Science have access to several HPC clusters. Each cluster is configured for various types and sizes of workflows.

			Back to [[Beginner's Guide to Computation in Statistical Research]]

	== Department Servers ==		== Department Servers ==

Ajshen at 06:14, 27 March 2026

2026-03-27T06:14:27Z

← Older revision		Revision as of 06:14, 27 March 2026
Line 15:		Line 15:
	\|Introduces basic Linux commands and instructions for connecting to department servers		\|Introduces basic Linux commands and instructions for connecting to department servers
	\|⭐⚫⚫⚫⚫		\|⭐⚫⚫⚫⚫
	\|style="background-color: #44ee88;" \| All server workflows		\|style="background-color: #44ee88;" \| '''All server workflows'''
	\|-		\|-
	\|[[R on the server]]		\|[[R on the server]]

Ajshen at 06:14, 27 March 2026

2026-03-27T06:14:18Z

← Older revision		Revision as of 06:14, 27 March 2026
Line 15:		Line 15:
	\|Introduces basic Linux commands and instructions for connecting to department servers		\|Introduces basic Linux commands and instructions for connecting to department servers
	\|⭐⚫⚫⚫⚫		\|⭐⚫⚫⚫⚫
	\|All server workflows		\|style="background-color: #44ee88;" \| All server workflows
	\|-		\|-
	\|[[R on the server]]		\|[[R on the server]]