Wright GPU Cluster - Revision history

Areinhar: /* Connecting and setting up your environment */ warn about no backups

2026-10-23T14:22:16Z

Connecting and setting up your environment: warn about no backups

← Older revision		Revision as of 14:22, 23 October 2026
Line 36:		Line 36:
	To upload scripts or data files, you can use <code>scp</code> from your local computer (Mac or Linux); on Windows, you can get the Tectia SSH graphical client [https://www.cmu.edu/computing/services/endpoint/software/access.html free through CMU].		To upload scripts or data files, you can use <code>scp</code> from your local computer (Mac or Linux); on Windows, you can get the Tectia SSH graphical client [https://www.cmu.edu/computing/services/endpoint/software/access.html free through CMU].

	You can~~ ~~also connect through the [https://code.visualstudio.com/docs/remote/ssh Remote SSH feature of VS Code] if you like. This is useful to~~ ~~		You can also connect through the [https://code.visualstudio.com/docs/remote/ssh Remote SSH feature of VS Code] if you like. This is useful to

	*navigate the file system from a proper GUI and		*navigate the file system from a proper GUI and
	*modify code/files/folder in place if you are not familiar with the command line		*modify code/files/folder in place if you are not familiar with the command line
	Files uploaded to your home directory will be accessible on the head node and all compute nodes, so they can be freely used in compute jobs.		Files uploaded to your home directory will be accessible on the head node and all compute nodes, so they can be freely used in compute jobs.

			=== Danger! Files are not backed up ===
			Wright's storage cluster is '''not backed up.''' You may lose data if a disk fails. Therefore:

			* Keep your code backed up in another location (such as by pushing it to GitHub regularly)
			* Do not use Wright for long-term storage of important data. Download local copies, push it to HuggingFace, or sync it to your Google Drive
			* Consider using an online backup tool like [https://www.borgbackup.org/ Borg] or [https://www.tarsnap.com/ Tarsnap] to regularly back up your home directory

	=== Using Conda ===		=== Using Conda ===

Jacobg2: /* Citing Wright */

2026-10-12T18:12:15Z

Citing Wright

← Older revision		Revision as of 18:12, 12 October 2026
Line 242:		Line 242:

	== Citing Wright ==		== Citing Wright ==
	If you use Wright in your research, please ~~consider citing~~ it with the text provided below...<syntaxhighlight lang="text">		If you use Wright in your research, please make sure to cite it with the text provided below...<syntaxhighlight lang="text">
	"This research was supported through the computational resources and staff contributions to the Wright HPC GPU cluster at Carnegie Mellon's Dietrich College of Humanities and Social Sciences provisioned and supported by the Department of Statistics & Data Science, the Department of Philosophy, The Carnegie Mellon Institute for Strategy and Technology, and the Dietrich College Dean's Office."		"This research was supported through the computational resources and staff contributions to the Wright HPC GPU cluster at Carnegie Mellon's Dietrich College of Humanities and Social Sciences provisioned and supported by the Department of Statistics & Data Science, the Department of Philosophy, The Carnegie Mellon Institute for Strategy and Technology, and the Dietrich College Dean's Office."
	</syntaxhighlight>		</syntaxhighlight>

Jacobg2: /* Citing Wright */

2026-10-12T18:11:05Z

Citing Wright

← Older revision		Revision as of 18:11, 12 October 2026
Line 242:		Line 242:

	== Citing Wright ==		== Citing Wright ==
	If you use Wright in your research, please consider citing it with the text provided below...		If you use Wright in your research, please consider citing it with the text provided below...<syntaxhighlight lang="text">
			"This research was supported through the computational resources and staff contributions to the Wright HPC GPU cluster at Carnegie Mellon's Dietrich College of Humanities and Social Sciences provisioned and supported by the Department of Statistics & Data Science, the Department of Philosophy, The Carnegie Mellon Institute for Strategy and Technology, and the Dietrich College Dean's Office."
			</syntaxhighlight>

Jacobg2 at 18:10, 12 October 2026

2026-10-12T18:10:16Z

← Older revision		Revision as of 18:10, 12 October 2026
Line 240:		Line 240:

	The scratch space is thus not for final model outputs or results. It is ideal for things like saving checkpoints while your job is running, dumping data to debug a job, saving output from one script to be immediately used by another script, etc.		The scratch space is thus not for final model outputs or results. It is ideal for things like saving checkpoints while your job is running, dumping data to debug a job, saving output from one script to be immediately used by another script, etc.

			== Citing Wright ==
			If you use Wright in your research, please consider citing it with the text provided below...

Jacobg2 at 18:08, 12 October 2026

2026-10-12T18:08:59Z

← Older revision		Revision as of 18:08, 12 October 2026
Line 1:		Line 1:

	~~[[index.php?title=Category:Wright Cluster]]~~		The department has access to a powerful GPU-based cluster named [[wikipedia:Sewall_Wright\|Wright]] which is hosted here at CMU. This is a 4-node and growing cluster that is a shared resource for Statistics & Data Science, Philosophy, and other departments in Dietrich College. Each department has priority access to its own compute node, but will be able to share the resources from other departments when not in use.

	The department has access to a powerful GPU-based cluster named [~~https~~:~~//en.wikipedia.org/wiki/~~Sewall_Wright Wright] which is hosted here at CMU. This is a 4-node and growing cluster that is a shared resource for Statistics & Data Science, Philosophy, and other departments in Dietrich College. Each department has priority access to its own compute node, but will be able to share the resources from other departments when not in use.

	== Access requests ==		== Access requests ==

Jacobg2: /* Cluster setup */

2026-10-12T13:12:11Z

Cluster setup

← Older revision		Revision as of 13:12, 12 October 2026
Line 1:		Line 1:

	[[Category:Wright Cluster]]		[[index.php?title=Category:Wright Cluster]]

	The department has access to a powerful GPU-based cluster named [https://en.wikipedia.org/wiki/Sewall_Wright Wright] which is hosted here at CMU. This is a 4-node and growing cluster that is a shared resource for Statistics & Data Science, Philosophy, and other departments in Dietrich College. Each department has priority access to its own compute node, but will be able to share the resources from other departments when not in use.		The department has access to a powerful GPU-based cluster named [https://en.wikipedia.org/wiki/Sewall_Wright Wright] which is hosted here at CMU. This is a 4-node and growing cluster that is a shared resource for Statistics & Data Science, Philosophy, and other departments in Dietrich College. Each department has priority access to its own compute node, but will be able to share the resources from other departments when not in use.
Line 14:		Line 14:
	*The compute nodes: Jobs run on these. Each has 256 CPU cores, 8 NVIDIA L40 GPU cards with 48 GB of video RAM each, approximately 1 TB of RAM, and a total of approximately 3 TB of local data/scratch space.		*The compute nodes: Jobs run on these. Each has 256 CPU cores, 8 NVIDIA L40 GPU cards with 48 GB of video RAM each, approximately 1 TB of RAM, and a total of approximately 3 TB of local data/scratch space.
	*The file server: Holds 50TB of storage. The head node and compute nodes connect via fiberoptic networking. Your home directory is hosted on the file server, so the same files are accessible on the head node and all compute nodes.		*The file server: Holds 50TB of storage. The head node and compute nodes connect via fiberoptic networking. Your home directory is hosted on the file server, so the same files are accessible on the head node and all compute nodes.
			**Statistics & Data Science accounts are limited to 5TB of space. More can be requested if required.
	The cluster is currently partitioned. One compute node is reserved for each department involved.		The cluster is currently partitioned. One compute node is reserved for each department involved.

Areinhar: fix category link

2026-08-21T14:24:45Z

fix category link

← Older revision		Revision as of 14:24, 21 August 2026
Line 1:		Line 1:

	[[~~index.php?title=~~Category:Wright Cluster]]		[[Category:Wright Cluster]]

	The department has access to a powerful GPU-based cluster named [https://en.wikipedia.org/wiki/Sewall_Wright Wright] which is hosted here at CMU. This is a 4-node and growing cluster that is a shared resource for Statistics & Data Science, Philosophy, and other departments in Dietrich College. Each department has priority access to its own compute node, but will be able to share the resources from other departments when not in use.		The department has access to a powerful GPU-based cluster named [https://en.wikipedia.org/wiki/Sewall_Wright Wright] which is hosted here at CMU. This is a 4-node and growing cluster that is a shared resource for Statistics & Data Science, Philosophy, and other departments in Dietrich College. Each department has priority access to its own compute node, but will be able to share the resources from other departments when not in use.

Areinhar: /* Job etiquette */ clarify prioritization

2026-08-21T14:23:59Z

Job etiquette: clarify prioritization

← Older revision		Revision as of 14:23, 21 August 2026
Line 1:		Line 1:

	[[Category:Wright Cluster]]		[[index.php?title=Category:Wright Cluster]]

	The department has access to a powerful GPU-based cluster named [https://en.wikipedia.org/wiki/Sewall_Wright Wright] which is hosted here at CMU. This is a 4-node and growing cluster that is a shared resource for Statistics & Data Science, Philosophy, and other departments in Dietrich College. Each department has priority access to its own compute node, but will be able to share the resources from other departments when not in use.		The department has access to a powerful GPU-based cluster named [https://en.wikipedia.org/wiki/Sewall_Wright Wright] which is hosted here at CMU. This is a 4-node and growing cluster that is a shared resource for Statistics & Data Science, Philosophy, and other departments in Dietrich College. Each department has priority access to its own compute node, but will be able to share the resources from other departments when not in use.
Line 91:		Line 91:
	Jobs are submitted to a partition. '''The main partitions will only accept jobs from users in the corresponding department''' -- jobs submitted to the wrong partition will be rejected. Partitions correspond to nodes, so this ensures each department's jobs use their own node.		Jobs are submitted to a partition. '''The main partitions will only accept jobs from users in the corresponding department''' -- jobs submitted to the wrong partition will be rejected. Partitions correspond to nodes, so this ensures each department's jobs use their own node.

	If you want to use other nodes, you can submit jobs to the <code>all</code> partition. This requests any available node from any partition, so your job will run on whichever node has resources available. However, your job will have lower priority, as described below; if your all job runs on the philosophy node, a user who submits to <code>phil</code> can preempt your job, causing it to be killed to make room for them, then restarted after their job is done.		If you want to use other nodes, you can submit jobs to the <code>all</code> partition. This requests any available node from any partition, so your job will run on whichever node has resources available. However, your job will have lower priority, as described below; if your all job runs on the philosophy node, a user who submits to <code>phil</code> can preempt your job, causing it to be killed to make room for them, then restarted after their job is done. See Job Etiquette below for when to use <code>all</code>.

	If you want to use a specific node within <code>all</code> you can specify the node within your submit script as shown below.<syntaxhighlight lang="batch">		If you want to use a specific node within <code>all</code> you can specify the node within your submit script as shown below.<syntaxhighlight lang="batch">
Line 102:		Line 102:
	Examples:		Examples:

	* You submit Job A to the <code>all</code> partition. Someone else submits Job B to <code>statds</code>. The <code>all</code> partition has lower QOS, so Job B will get to use the StatDS node first, and Job A will either wait or use a different node.
	* You submit Job A to the <code>all</code> partition. It begins to run on the philosophy node. Someone else submits Job B to <code>phil</code>, and there are insufficient resources to run both at the same time. Job A is killed, Job B runs, and Job A is placed back into the queue to be run when resources are available.
	* Someone submits Job A that requests 6 GPUs. Later, you submit Job B that requests 4 GPUs on the same partition. The compute node only has 8 GPUs, so Job B will be queued until Job A completes.		* Someone submits Job A that requests 6 GPUs. Later, you submit Job B that requests 4 GPUs on the same partition. The compute node only has 8 GPUs, so Job B will be queued until Job A completes.
	* Someone submits Job A that requests 6 GPUs. Later, you submit Job B that requests 2 GPUs. Job B can run immediately.		* Someone submits Job A that requests 6 GPUs. Later, you submit Job B that requests 2 GPUs. Job B can run immediately.
	* You submit a job that requires all 8 GPUs on a node, but 4 are currently in use. Your job is put in the queue and scheduled to run when Slurm knows the current jobs will finish (because of their maximum time limit). Meanwhile, someone submits a job using 2 GPUs and a short time limit. Slurm lets it run immediately, because it knows it will be done before your job is scheduled to run. Other jobs may be put off until after yours runs, so yours can get the 8 GPUs it needs.		* You submit a job that requires all 8 GPUs on a node, but 4 are currently in use. Your job is put in the queue and scheduled to run when Slurm knows the current jobs will finish (because of their maximum time limit). Meanwhile, someone submits a job using 2 GPUs and a short time limit. Slurm lets it run immediately, because it knows it will be done before your job is scheduled to run. Other jobs may be put off until after yours runs, so yours can get the 8 GPUs it needs.
	* You submit 50 jobs that each require 1 GPU. The node has only 8 GPUs, but several are in use. Several of your jobs run, but the others are queued for later. The fair-sharing system assigns them a lower priority, because you have run so many jobs, and jobs submitted by other users will run first.		* You submit 50 jobs that each require 1 GPU. The node has only 8 GPUs, but several are in use. Several of your jobs run, but the others are queued for later. The fair-sharing system assigns them a lower priority, because you have run so many jobs, and jobs submitted by other users will run first.
			* You submit Job A to the <code>all</code> partition. Someone else submits Job B to <code>statds</code>. The <code>all</code> partition has lower QOS, so Job B will get to use the StatDS node first, and Job A will either wait or use a different node.
			* You submit Job A to the <code>all</code> partition. It begins to run on the philosophy node. Someone else submits Job B to <code>phil</code>, and there are insufficient resources to run both at the same time. Job A is killed, Job B runs, and Job A is placed back into the queue to be run when resources are available.
	You can view the currently running or queued jobs by running <code>squeue</code> on the head node, as described above. The <code>NODELIST(REASON)</code> column will indicate which jobs are queued by giving special codes:		You can view the currently running or queued jobs by running <code>squeue</code> on the head node, as described above. The <code>NODELIST(REASON)</code> column will indicate which jobs are queued by giving special codes:

Line 117:		Line 117:

	* Request the minimum resources required for your job, so other users can access other resources.		* Request the minimum resources required for your job, so other users can access other resources.
			* For ordinary one-off jobs, use your department's partition so you get top priority for your node.
	* If you need to submit a large number of jobs and aren't in a hurry, consider using the <code>all</code> partition. Using <code>all</code> lets you get all available resources, but at lower priority, so other users can run jobs as well and you don't hog the cluster.		* If you need to submit a large number of jobs and aren't in a hurry, consider using the <code>all</code> partition. Using <code>all</code> lets you get all available resources, but at lower priority, so other users can run jobs as well and you don't hog the cluster.
	* Watch your disk space usage. With large deep models it is easy to use many terabytes of storage, and each department has a quota. [https://util.stat.cmu.edu/quotareport/ You can check the quota and your own usage here.]		* Watch your disk space usage. With large deep models it is easy to use many terabytes of storage, and each department has a quota. [https://util.stat.cmu.edu/quotareport/ You can check the quota and your own usage here.]

	=== Job ~~Limits~~ ===		=== Job limits ===
	There is no practical limit to the number of jobs you can have in the queue at a time (there is a maximum size limit for job arrays that can be checked via <code>scontrol show config \| grep MaxArraySize</code>. However, in order to prevent resource monopolization, '''no user ~~will~~ have more than 6 jobs running at once on the <code>statds</code> partition.''' If a user submits more than 6 jobs to the <code>statds</code> partition, the excess jobs will sit in the queue until one or more of the first 6 are completed. '''This limit does not apply to the <code>all</code> partition''', so we encourage users with a large number of jobs to use the <code>all</code> partition instead of the <code>statds</code> partition.		There is no practical limit to the number of jobs you can have in the queue at a time (there is a maximum size limit for job arrays that can be checked via <code>scontrol show config \| grep MaxArraySize</code>. However, in order to prevent resource monopolization, '''no user can have more than 6 jobs running at once on the <code>statds</code> partition.''' If a user submits more than 6 jobs to the <code>statds</code> partition, the excess jobs will sit in the queue until one or more of the first 6 are completed. '''This limit does not apply to the <code>all</code> partition''', so we encourage users with a large number of jobs to use the <code>all</code> partition instead of the <code>statds</code> partition.

	== Setting up jobs ==		== Setting up jobs ==

Ajshen: /* Job Limits */

2026-07-01T18:28:32Z

Job Limits

← Older revision		Revision as of 18:28, 1 July 2026
Line 121:		Line 121:

	=== Job Limits ===		=== Job Limits ===
	There is no practical limit to the number of jobs you can have in the queue at a time (there is a maximum size limit for job arrays that can be checked via scontrol show config \| grep MaxArraySize. However, in order to prevent resource monopolization, '''no user will have more than 6 jobs at once ~~will running~~ on the statds partition.''' If a user submits more than 6 jobs to the statds partition, the excess jobs will sit in the queue until one or more of the first 6 are completed. '''This limit does not apply to the all partition''', so we encourage users with a large number of jobs to use the all partition instead of the statds partition.		There is no practical limit to the number of jobs you can have in the queue at a time (there is a maximum size limit for job arrays that can be checked via <code>scontrol show config \| grep MaxArraySize</code>. However, in order to prevent resource monopolization, '''no user will have more than 6 jobs running at once on the <code>statds</code> partition.''' If a user submits more than 6 jobs to the <code>statds</code> partition, the excess jobs will sit in the queue until one or more of the first 6 are completed. '''This limit does not apply to the <code>all</code> partition''', so we encourage users with a large number of jobs to use the <code>all</code> partition instead of the <code>statds</code> partition.

	== Setting up jobs ==		== Setting up jobs ==

Ajshen: /* Partitions */

2026-07-01T18:27:10Z

Partitions

← Older revision		Revision as of 18:27, 1 July 2026
Line 119:		Line 119:
	* If you need to submit a large number of jobs and aren't in a hurry, consider using the <code>all</code> partition. Using <code>all</code> lets you get all available resources, but at lower priority, so other users can run jobs as well and you don't hog the cluster.		* If you need to submit a large number of jobs and aren't in a hurry, consider using the <code>all</code> partition. Using <code>all</code> lets you get all available resources, but at lower priority, so other users can run jobs as well and you don't hog the cluster.
	* Watch your disk space usage. With large deep models it is easy to use many terabytes of storage, and each department has a quota. [https://util.stat.cmu.edu/quotareport/ You can check the quota and your own usage here.]		* Watch your disk space usage. With large deep models it is easy to use many terabytes of storage, and each department has a quota. [https://util.stat.cmu.edu/quotareport/ You can check the quota and your own usage here.]

			=== Job Limits ===
			There is no practical limit to the number of jobs you can have in the queue at a time (there is a maximum size limit for job arrays that can be checked via scontrol show config \| grep MaxArraySize. However, in order to prevent resource monopolization, '''no user will have more than 6 jobs at once will running on the statds partition.''' If a user submits more than 6 jobs to the statds partition, the excess jobs will sit in the queue until one or more of the first 6 are completed. '''This limit does not apply to the all partition''', so we encourage users with a large number of jobs to use the all partition instead of the statds partition.

	== Setting up jobs ==		== Setting up jobs ==