Linux and department servers - Revision history

Ajshen at 21:32, 24 August 2026

2026-08-24T21:32:27Z

← Older revision		Revision as of 21:32, 24 August 2026
Line 2:		Line 2:
	[[Category:Remote Server Access]]		[[Category:Remote Server Access]]

	{{~~ServerSidebar~~}}		{{ScalingSidebar}}

	The department has two sets of servers, the Leviathans and the Hydras, that can help you run large analyses and gain a better understanding of common command line structure for more high scale computing.		The department has two sets of servers, the Leviathans and the Hydras, that can help you run large analyses and gain a better understanding of common command line structure for more high scale computing.

Ajshen at 21:31, 24 August 2026

2026-08-24T21:31:55Z

Show changes

Jacobg2: /* Checking current usage */

2026-08-06T13:04:56Z

Checking current usage

← Older revision		Revision as of 13:04, 6 August 2026
Line 148:		Line 148:

	=== Checking current usage===		=== Checking current usage===
	To check which servers are currently being used and which are open, see the [https://~~info~~.stat.cmu.edu/grafana/d/beuw1q6y3dgjkc/hydra-cluster?orgId=1 Grafana dashboard for the hydras] and [https://~~info~~.stat.cmu.edu/grafana/d/asjkldfbnvajdsfh213/leviathan-cluster?orgId=1 for the Leviathans]. It lists them all and their current usage.		To check which servers are currently being used and which are open, see the [https://util.stat.cmu.edu/grafana/d/beuw1q6y3dgjkc/hydra-cluster?orgId=1&from=now-6h&to=now&timezone=browser Grafana dashboard for the hydras] and [https://util.stat.cmu.edu/grafana/d/asjkldfbnvajdsfh213/leviathan-cluster?orgId=1&from=now-6h&to=now&timezone=browser for the Leviathans]. It lists them all and their current usage.

	When you're logged in to one node, you can use the <code>top</code> command. <code>top</code> displays the current usage of the server’s resources with a list of jobs run. It also tells you the percentage of the cores that are being used and the amount of RAM (memory) that is available. See [https://www.howtogeek.com/668986/how-to-use-the-linux-top-command-and-understand-its-output/ howtogeek] resources on how to interpret it all.		When you're logged in to one node, you can use the <code>top</code> command. <code>top</code> displays the current usage of the server’s resources with a list of jobs run. It also tells you the percentage of the cores that are being used and the amount of RAM (memory) that is available. See [https://www.howtogeek.com/668986/how-to-use-the-linux-top-command-and-understand-its-output/ howtogeek] resources on how to interpret it all.

Areinhar: formatting tweaks

2026-07-07T14:22:28Z

formatting tweaks

← Older revision		Revision as of 14:22, 7 July 2026
Line 34:		Line 34:
	If you’re accessing the servers frequently, we highly recommend setting up passwordless SSH (see below).		If you’re accessing the servers frequently, we highly recommend setting up passwordless SSH (see below).

	=== Passwordless ~~ssh~~ ===		=== Passwordless SSH ===
	To connect to the server without typing in your password every time, you need to generate an SSH key pair. This creates a private key that stays securely on your laptop, and a public key that the servers use to recognize you.		To connect to the server without typing in your password every time, you need to generate an SSH key pair. This creates a private key that stays securely on your laptop, and a public key that the servers use to recognize you.

Line 170:		Line 170:
	# Do stuff in my_data		# Do stuff in my_data
	</syntaxhighlight>It is also very important to remember that <code>/scratch</code> '''is specific to each Leviathan and Ghidorah and will not follow you between servers'''. Thus, you should not use <code>/scratch</code> for long-term storage or for important data. In addition, space on <code>/scratch</code> is limited, so be sure to clean up your data once you don't need it anymore.		</syntaxhighlight>It is also very important to remember that <code>/scratch</code> '''is specific to each Leviathan and Ghidorah and will not follow you between servers'''. Thus, you should not use <code>/scratch</code> for long-term storage or for important data. In addition, space on <code>/scratch</code> is limited, so be sure to clean up your data once you don't need it anymore.


	==Running jobs==		==Running jobs==

Ajshen at 03:38, 30 June 2026

2026-06-30T03:38:42Z

Show changes

Ajshen: /* What resources exist */

2026-06-30T03:31:53Z

What resources exist

← Older revision		Revision as of 03:31, 30 June 2026
Line 8:		Line 8:
	== What resources exist ==		== What resources exist ==

	# The department provides 11 leviathan servers (named {<code>leviathan1</code>, ..., <code>leviathan11</code>}<code>.stat.cmu.edu</code>)~~. Each Leviathan has 16 CPU cores and 128 GB of RAM. The Leviathan CPUs are much faster than the former Hydra CPUs~~.		# The department provides 11 leviathan servers (named {<code>leviathan1</code>, ..., <code>leviathan11</code>}<code>.stat.cmu.edu</code>).
	# The department provides 3 Ghidorah servers (named {<code>ghidorah1</code>, ..., <code>ghidorah3</code>}<code>.stat.cmu.edu</code>)~~. Each Ghidorah has 64GB of RAM, 32 core processors and NVIDIA RTX 4090 GPU. Note that ghidorah3 is reserved for MSCF student capstone projects in the fall~~.		# The department provides 3 Ghidorah GPU servers (named {<code>ghidorah1</code>, ..., <code>ghidorah3</code>}<code>.stat.cmu.edu</code>).
	# The Ghidorahs and Leviathans all share the same data file system. This allows you to upload a file to any of the servers and then have it on all of them.		# The Ghidorahs and Leviathans all share the same data file system. This allows you to upload a file to any of the servers and then have it on all of them.
	~~#[[OpenMPI]] is installed~~
	~~#*This allows MPI based parallelization to be used~~
	~~#* Note that the hydras ARE NOT network connected, so MPI can only be used within a single hydra. It cannot communicate across multiple hydras.~~
	~~#*If you need to run something in parallel across multiple nodes, you should use [[PSC]] instead.~~

	''See [[what_resources_when\| What resources when]] for guides to help decide which resource you should be using.''		''See [[what_resources_when\| What resources when]] for guides to help decide which resource you should be using.''

Ajshen at 02:43, 29 June 2026

2026-06-29T02:43:10Z

← Older revision		Revision as of 02:43, 29 June 2026
Line 1:		Line 1:
	[[Category:Department Servers]]		[[Category:Department Servers]]
			[[Category:Remote Server Access]]

	The department has two sets of servers, the Leviathans and the Hydras, that can help you run large analyses and gain a better understanding of common command line structure for more high scale computing. One can also view the department servers a bit more as a “sandbox” or “playground” as there are very few constraints and it’s okay to not run as polished code as you might on the [[PSC]] or Azure.		The department has two sets of servers, the Leviathans and the Hydras, that can help you run large analyses and gain a better understanding of common command line structure for more high scale computing. One can also view the department servers a bit more as a “sandbox” or “playground” as there are very few constraints and it’s okay to not run as polished code as you might on the [[PSC]] or Azure.

Ajshen at 21:21, 28 June 2026

2026-06-28T21:21:00Z

← Older revision		Revision as of 21:21, 28 June 2026
Line 1:		Line 1:
			[[Category:Department Servers]]


	The department has two sets of servers, the Leviathans and the Hydras, that can help you run large analyses and gain a better understanding of common command line structure for more high scale computing. One can also view the department servers a bit more as a “sandbox” or “playground” as there are very few constraints and it’s okay to not run as polished code as you might on the [[PSC]] or Azure.		The department has two sets of servers, the Leviathans and the Hydras, that can help you run large analyses and gain a better understanding of common command line structure for more high scale computing. One can also view the department servers a bit more as a “sandbox” or “playground” as there are very few constraints and it’s okay to not run as polished code as you might on the [[PSC]] or Azure.

Ajshen at 21:16, 28 June 2026

2026-06-28T21:16:39Z

@@ Line 15: / Line 15: @@
 ''See [[what_resources_when| What resources when]] for guides to help decide which resource you should be using.''
 ==Introduction to connecting to the server and useful Linux commands==
@@ Line 184: / Line 150: @@
 '''TODO: add information about <code>sed</code>, <code>chmod</code>, and a link to bash files.'''
 ==Text editors==

Ajshen: /* Using the servers respectfully */

2026-04-02T09:56:12Z

Using the servers respectfully

← Older revision		Revision as of 09:56, 2 April 2026
Line 22:		Line 22:
	#Do not use '''all''' the cores on a server, and do not use '''all''' the servers at the same time, so others can get work done.		#Do not use '''all''' the cores on a server, and do not use '''all''' the servers at the same time, so others can get work done.
	#If you have a long-running job, '''deprioritize it''' so others can run too.		#If you have a long-running job, '''deprioritize it''' so others can run too.
			#Use <code>/scratch</code> for jobs that require intense disk I/O operations

	Each of these is easy to check.		Each of these is easy to check.

	===Checking current usage===		=== Checking current usage===
	To check which servers are currently being used and which are open, see the [https://info.stat.cmu.edu/grafana/d/beuw1q6y3dgjkc/hydra-cluster?orgId=1 Grafana dashboard for the hydras] and [https://info.stat.cmu.edu/grafana/d/asjkldfbnvajdsfh213/leviathan-cluster?orgId=1 for the Leviathans]. It lists them all and their current usage.		To check which servers are currently being used and which are open, see the [https://info.stat.cmu.edu/grafana/d/beuw1q6y3dgjkc/hydra-cluster?orgId=1 Grafana dashboard for the hydras] and [https://info.stat.cmu.edu/grafana/d/asjkldfbnvajdsfh213/leviathan-cluster?orgId=1 for the Leviathans]. It lists them all and their current usage.

Line 33:		Line 34:
	We recommend that when using parallel processing, use fewer than the total number of cores on the server. Most parallel processing packages let you choose the number of cores to use; the hydras have 12, so pick something smaller.		We recommend that when using parallel processing, use fewer than the total number of cores on the server. Most parallel processing packages let you choose the number of cores to use; the hydras have 12, so pick something smaller.

	===Deprioritize when possible===		=== Deprioritize when possible===
	<code>nice/renice</code> allows you to lower the priority of your jobs, so the operating system lets other processes run first. See [https://www.howtogeek.com/411979/how-to-set-process-priorities-with-the-nice-and-renice-commands-in-linux/ howtogeek] resources on the basics. <code>nice</code> sets the priority, and <code>renice</code> updates your priority. On the department servers, everyone starts at the same priority, and you can only '''deprioritize''' your jobs -- you can't make them higher priority than everyone else.<syntaxhighlight lang="bash">nice -20 Rscript some-big-script.R		<code>nice/renice</code> allows you to lower the priority of your jobs, so the operating system lets other processes run first. See [https://www.howtogeek.com/411979/how-to-set-process-priorities-with-the-nice-and-renice-commands-in-linux/ howtogeek] resources on the basics. <code>nice</code> sets the priority, and <code>renice</code> updates your priority. On the department servers, everyone starts at the same priority, and you can only '''deprioritize''' your jobs -- you can't make them higher priority than everyone else.<syntaxhighlight lang="bash">nice -20 Rscript some-big-script.R
	renice -n 20 processid # get the process ID from top</syntaxhighlight>		renice -n 20 processid # get the process ID from top</syntaxhighlight>

			===Using <code>/scratch</code>===
			On department servers, your home directory (the default location you start in when you ssh) is not actually located on the physical server itself. Instead, every department server uses network attached storage (NAS), where everybody's home directly lives on a central storage server and the individual Leviathans and Ghidorahs connect to that storage server. This is the functionality that lets your files follow you no matter what node you log into.

			While computing processes on one server do not affect processes on other servers, every server shares the same NAS. Therefore, '''if a user is reading and writing a ton of data to their home directory, it will slow down reading and writing for everyone, even if you aren't on the same server as this user.''' Even if your job does not read or write much data, disk I/O slowdowns will affect common operations, such as ssh, importing packages, and uploading/downloading files.

			If you need to run a job that requires a ton of reading and writing data to disk, '''do not read/write using your home directory'''. Instead, use the <code>/scratch</code> folder located on every server. This folder points to an SSD that is specific to that server. Not only will read and writes to <code>/scratch</code> not affect users on other servers, but it will also be generally faster than using your home directory even if nobody else is online. '''If you plan to use''' <code>/scratch</code>''', you should make a new folder inside of''' <code>/scratch</code> '''first and put your data inside that folder. For example:'''<syntaxhighlight lang="bash">
			cd /scratch
			mkdir my_data # Default permission allows only you to modify contents
			cd my_data
			# Do stuff in my_data
			</syntaxhighlight>It is also very important to remember that <code>/scratch</code> '''is specific to each Leviathan and Ghidorah and will not follow you between servers'''. Thus, you should not use <code>/scratch</code> for long-term storage or for important data. In addition, space on <code>/scratch</code> is limited, so be sure to clean up your data once you don't need it anymore.

	==Introduction to connecting to the server and useful Linux commands==		==Introduction to connecting to the server and useful Linux commands==

Line 61:		Line 75:

	*For Linux or Mac, open up a terminal and check whether you already have a key pair: run <code>ls ~/.ssh</code> and see if you have any files with names starting <code>id_</code>. If you do, you already have a key pair, and you can skip to the <code>ssh-copy-id</code> step below.		*For Linux or Mac, open up a terminal and check whether you already have a key pair: run <code>ls ~/.ssh</code> and see if you have any files with names starting <code>id_</code>. If you do, you already have a key pair, and you can skip to the <code>ssh-copy-id</code> step below.
	*Next, generate a key pair (both public and private key). There are different types of keys (RSA, ECDSA, and Ed25519). The <code>-t</code> option specifies the type of key pair.		* Next, generate a key pair (both public and private key). There are different types of keys (RSA, ECDSA, and Ed25519). The <code>-t</code> option specifies the type of key pair.
	ssh-keygen -t ed25519		ssh-keygen -t ed25519
	*A prompt for what to name the key pair and what folder to place it in should appear. The username should be the username for your local computer and <code>id_ed25519</code> is the key pair name. You can change it. The local key should be stored in the <code>~/.ssh</code> folder on your local computer.		*A prompt for what to name the key pair and what folder to place it in should appear. The username should be the username for your local computer and <code>id_ed25519</code> is the key pair name. You can change it. The local key should be stored in the <code>~/.ssh</code> folder on your local computer.
Line 69:		Line 83:
	*A prompt for a passphrase should appear. This is for securing the local key (on local computer) and is optional. <code>Enter Passphrase (empty for no passphrase):</code>		*A prompt for a passphrase should appear. This is for securing the local key (on local computer) and is optional. <code>Enter Passphrase (empty for no passphrase):</code>

	* Make sure there is a <code>~/.ssh</code> folder on the server. If not, make a .ssh folder.		*Make sure there is a <code>~/.ssh</code> folder on the server. If not, make a .ssh folder.
	*Next, upload the public key to remote server. This command will upload the public key to all the hydras and ghidorahs since they share a file system.		*Next, upload the public key to remote server. This command will upload the public key to all the hydras and ghidorahs since they share a file system.
	ssh-copy-id andrew_id@leviathan1.stat.cmu.edu		ssh-copy-id andrew_id@leviathan1.stat.cmu.edu
Line 77:		Line 91:
	You should now be able to SSH into the server without being prompted for a password.		You should now be able to SSH into the server without being prompted for a password.

	===File sharing (<code>scp</code> +) ===		===File sharing (<code>scp</code> +)===

	There is also command line code to copy files from your personal computer to the server. This is done with a SCP (secure copy) tool. There are also GUIs (graphical user interfaces) that allow you to “drag and drop” files from your computer to the server (and visa-versa). Currently, a lot of students use GUIs to deal with file sharing.		There is also command line code to copy files from your personal computer to the server. This is done with a SCP (secure copy) tool. There are also GUIs (graphical user interfaces) that allow you to “drag and drop” files from your computer to the server (and visa-versa). Currently, a lot of students use GUIs to deal with file sharing.
Line 131:		Line 145:
	</li>		</li>
	</ol>		</ol>
	===Basic CLI (Linux command line): <code>bash</code>===		=== Basic CLI (Linux command line): <code>bash</code>===

	Understanding the basic Linux command line is important to be able to navigate when you are actually on the server. Below are some based commands to move around the file system, manipulate files, and view storage structure.		Understanding the basic Linux command line is important to be able to navigate when you are actually on the server. Below are some based commands to move around the file system, manipulate files, and view storage structure.
Line 146:		Line 160:
	\|		\|
	<code>cd _dir_</code>		<code>cd _dir_</code>
	\|change directory		\| change directory
	\|-		\|-
	\|		\|
	<code>rm _file_</code>		<code>rm _file_</code>
	\|remove / delete file		\|remove / delete file
	\|-		\|-
	\|		\|
	<code>mv _file_ _dir_</code>		<code>mv _file_ _dir_</code>
	\|move file (can change name)		\|move file (can change name)
Line 182:		Line 196:
	We have included thoughts about running jobs in two places.		We have included thoughts about running jobs in two places.

	#The first looks at tools that allow you to keep running jobs on the server when you close the connection to the server (turn off your laptop, etc.). These tools are call [[~~Terminal_multiplexers\|~~terminal multiplexers]].		# The first looks at tools that allow you to keep running jobs on the server when you close the connection to the server (turn off your laptop, etc.). These tools are call [[terminal multiplexers]].
	#The second set of tools focus on how to run your [[R_on_the_server\|R analysis]] or [[Python_on_the_server\|Python analysis]] code on the server.		#The second set of tools focus on how to run your [[R_on_the_server\|R analysis]] or [[Python_on_the_server\|Python analysis]] code on the server.