Scratch Storage - Revision history

Ajshen at 03:34, 29 June 2026

2026-06-29T03:34:02Z

← Older revision		Revision as of 03:34, 29 June 2026
Line 1:		Line 1:

			[[Category:Remote Server Access]]
			[[Category:Big Data Tools]]
			[[Category:Department Servers]]
			[[Category:Wright Cluster]]
			[[Category:PSC]]

	Scratch storage is a location where you can store data directly on the machine. These locations use faster flash based storage, which is better suited for the heavy reading and writing of data.		Scratch storage is a location where you can store data directly on the machine. These locations use faster flash based storage, which is better suited for the heavy reading and writing of data.

Jacobg2: /* Use Cases */

2026-04-02T15:52:32Z

Use Cases

← Older revision		Revision as of 15:52, 2 April 2026
Line 11:		Line 11:

	== Use Cases ==		== Use Cases ==
	Scratch storage is useful for temporarily storing large files you will be reading and writing from ~~heavily~~. ~~Because the scratch storage~~ is ~~separate between each machine~~, ~~not shared, and because it is limited in size, it is less useful for long-term storage~~.		Scratch storage is useful for temporarily storing large files you will be reading and writing from. Once your job is completed, you can move the data over to the NAS (/home/yourusername).

	Good uses include:		Good uses include:
Line 19:		Line 19:
	* Writing huge outputs from scripts when you're still debugging them and won't need to keep the output for long		* Writing huge outputs from scripts when you're still debugging them and won't need to keep the output for long

	~~Of course, anything~~ saved to <code>/scratch</code> can be moved (with <code>cp</code> or <code>mv</code>) to your home directory if you decide to keep it permanently.		Anything saved to <code>/scratch</code> can be moved (with <code>cp</code> or <code>mv</code>) to your home directory if you decide to keep it permanently.

	== Scratch Storage Location on Each Leviathan ==		== Scratch Storage Location on Each Leviathan ==

Areinhar: expand with use cases

2026-04-02T14:40:07Z

expand with use cases

← Older revision		Revision as of 14:40, 2 April 2026
Line 1:		Line 1:
	Scratch storage is a location where you can store data directly on the machine. These locations use faster flash based storage, which is better suited for the heavy reading and writing of data.		Scratch storage is a location where you can store data directly on the machine. These locations use faster flash based storage, which is better suited for the heavy reading and writing of data.

	Every time you read/write from your "/home/yourusername" directory, you are accessing a remote storage location. This is slower to begin with, especially if other users are running read/write heavy jobs.		Every time you read/write from your <code>/home/yourusername</code> directory, you are accessing a remote storage location over the network. This is slower to begin with, especially if other users are running read/write heavy jobs.

	~~'''Storage Terminology'''~~

			== Storage Terminology ==
	NVME SSD - Fast flash based storage. Smaller available storage size.		NVME SSD - Fast flash based storage. Smaller available storage size.

Line 11:		Line 10:
	Each Leviathan is equipped with a 4TB NVME SSD (faster than the NAS which uses spinning disks.). This allows for heavy reading and writing at high speeds. Utilizing this space can speed up your jobs and help with congestion on the NAS.		Each Leviathan is equipped with a 4TB NVME SSD (faster than the NAS which uses spinning disks.). This allows for heavy reading and writing at high speeds. Utilizing this space can speed up your jobs and help with congestion on the NAS.

	====== Scratch Storage ~~location~~ on Each Leviathan ~~====~~==		== Use Cases ==
	/scratch		Scratch storage is useful for temporarily storing large files you will be reading and writing from heavily. Because the scratch storage is separate between each machine, not shared, and because it is limited in size, it is less useful for long-term storage.

			Good uses include:

			* Saving checkpoints or output from long-running jobs or simulations, to be analyzed and moved to long-term storage once the job is done
			* Saving output from one script that will immediately be used by another script
			* Writing huge outputs from scripts when you're still debugging them and won't need to keep the output for long

			Of course, anything saved to <code>/scratch</code> can be moved (with <code>cp</code> or <code>mv</code>) to your home directory if you decide to keep it permanently.

			== Scratch Storage Location on Each Leviathan ==
			Scratch storage is available in <code>/scratch</code>.

	You can view the amount of space used at /scratch for each leviathan [https://info.stat.cmu.edu/grafana/d/asjkldfbnvajdsfh213/leviathan-cluster?orgId=1 here].		You can view the amount of space used at <code>/scratch</code> for each leviathan [https://info.stat.cmu.edu/grafana/d/asjkldfbnvajdsfh213/leviathan-cluster?orgId=1 here].

	Every /scratch location on each leviathan is it's own storage location. Meaning if you create data on Leviathan1 on /scratch, it will not be ~~acessable~~ on Leviathan7 at /scratch.		Every <code>/scratch</code> location on each leviathan is it's own storage location. Meaning if you create data on Leviathan1 on <code>/scratch</code>, it will not be accessible on Leviathan7 at <code>/scratch</code>.

	====== Steps to use Scratch Storage ======		====== Steps to use Scratch Storage ======

	# Make a folder in /scratch location.		# Make a folder in /scratch location.
	#* mkdir /scratch/~~foldern1~~		#* <code>mkdir /scratch/yourusername</code>
	# Change ownership of the folder to you.		# Change ownership of the folder to you.
	#* chown ~~$USER~~:~~$USER~~ /scratch/~~folder1~~		#* <code>chown yourusername:yourusername /scratch/yourusername</code>
	# Update permissions on folder (modify as needed.)		# Update permissions on folder so only you can read it (modify as needed.)
	#* chmod 750 /scratch/~~folder1~~		#* <code>chmod 750 /scratch/yourusername</code>

	[[File:ScratchSpace.png\|center\|frame]]		[[File:ScratchSpace.png\|center\|frame]]

Jacobg2: /* Steps to use Scratch Storage */

2026-03-24T14:54:01Z

Steps to use Scratch Storage

← Older revision		Revision as of 14:54, 24 March 2026
Line 9:		Line 9:
	Spinning Disk Storage - Slow mechanical hard drive with spinning disks. Larger available storage size		Spinning Disk Storage - Slow mechanical hard drive with spinning disks. Larger available storage size

	Each Leviathan is equipped with a 4TB NVME SSD (faster than the NAS). This allows for heavy reading and writing at high speeds. Utilizing this space can speed up your jobs, and help with congestion on the NAS.		Each Leviathan is equipped with a 4TB NVME SSD (faster than the NAS which uses spinning disks.). This allows for heavy reading and writing at high speeds. Utilizing this space can speed up your jobs and help with congestion on the NAS.

	====== Scratch Storage location on Each Leviathan ======		====== Scratch Storage location on Each Leviathan ======
Line 21:		Line 21:

	# Make a folder in /scratch location.		# Make a folder in /scratch location.
	#* ~~sudo~~ mkdir /scratch/foldern1		#* mkdir /scratch/foldern1
	# Change ownership of the folder to you.		# Change ownership of the folder to you.
	#* ~~sudo~~ chown $USER:$USER /scratch/folder1		#* chown $USER:$USER /scratch/folder1
	# Update permissions on folder (modify as needed.)		# Update permissions on folder (modify as needed.)
	## chmod 750 /scratch/folder1		#* chmod 750 /scratch/folder1

	[[File:ScratchSpace.png\|center\|frame]]		[[File:ScratchSpace.png\|center\|frame]]

Jacobg2: /* Scratch Storage location on Each Leviathan */

2026-03-23T16:58:40Z

Scratch Storage location on Each Leviathan

← Older revision		Revision as of 16:58, 23 March 2026
Line 15:		Line 15:

	You can view the amount of space used at /scratch for each leviathan [https://info.stat.cmu.edu/grafana/d/asjkldfbnvajdsfh213/leviathan-cluster?orgId=1 here].		You can view the amount of space used at /scratch for each leviathan [https://info.stat.cmu.edu/grafana/d/asjkldfbnvajdsfh213/leviathan-cluster?orgId=1 here].

			Every /scratch location on each leviathan is it's own storage location. Meaning if you create data on Leviathan1 on /scratch, it will not be acessable on Leviathan7 at /scratch.

	====== Steps to use Scratch Storage ======		====== Steps to use Scratch Storage ======

Jacobg2: /* Scratch Storage location on Each Leviathan */

2026-03-23T16:57:03Z

Scratch Storage location on Each Leviathan

← Older revision		Revision as of 16:57, 23 March 2026
Line 8:		Line 8:

	Spinning Disk Storage - Slow mechanical hard drive with spinning disks. Larger available storage size		Spinning Disk Storage - Slow mechanical hard drive with spinning disks. Larger available storage size



	Each Leviathan is equipped with a 4TB NVME SSD (faster than the NAS). This allows for heavy reading and writing at high speeds. Utilizing this space can speed up your jobs, and help with congestion on the NAS.		Each Leviathan is equipped with a 4TB NVME SSD (faster than the NAS). This allows for heavy reading and writing at high speeds. Utilizing this space can speed up your jobs, and help with congestion on the NAS.
Line 15:		Line 13:
	====== Scratch Storage location on Each Leviathan ======		====== Scratch Storage location on Each Leviathan ======
	/scratch		/scratch

			You can view the amount of space used at /scratch for each leviathan [https://info.stat.cmu.edu/grafana/d/asjkldfbnvajdsfh213/leviathan-cluster?orgId=1 here].

	====== Steps to use Scratch Storage ======		====== Steps to use Scratch Storage ======

Jacobg2 at 14:25, 23 March 2026

2026-03-23T14:25:55Z

← Older revision		Revision as of 14:25, 23 March 2026
Line 1:		Line 1:
	~~Every time you read/write from your "/home/yourusername" directory, you are accessing~~ a ~~remote storage~~ location. ~~This means slowdowns can occur~~, ~~especially if other users are doing~~ the ~~same~~.		Scratch storage is a location where you can store data directly on the machine. These locations use faster flash based storage, which is better suited for the heavy reading and writing of data.

			Every time you read/write from your "/home/yourusername" directory, you are accessing a remote storage location. This is slower to begin with, especially if other users are running read/write heavy jobs.

			'''Storage Terminology'''
	'''~~Terms~~'''

	NVME SSD - Fast flash based storage. Smaller available storage size.		NVME SSD - Fast flash based storage. Smaller available storage size.

	Spinning Disk Storage - Slow ~~machanical~~ hard drive with spinning disks. Larger available storage size		Spinning Disk Storage - Slow mechanical hard drive with spinning disks. Larger available storage size



	Each Leviathan is equipped with a 4TB NVME SSD (faster than the NAS). This allows for heavy reading and writing at high speeds. Utilizing this space can speed up your jobs, and help with congestion on the NAS.		Each Leviathan is equipped with a 4TB NVME SSD (faster than the NAS). This allows for heavy reading and writing at high speeds. Utilizing this space can speed up your jobs, and help with congestion on the NAS.

			====== Scratch Storage location on Each Leviathan ======
			/scratch

			====== Steps to use Scratch Storage ======

			# Make a folder in /scratch location.
			#* sudo mkdir /scratch/foldern1
			# Change ownership of the folder to you.
			#* sudo chown $USER:$USER /scratch/folder1
			# Update permissions on folder (modify as needed.)
			## chmod 750 /scratch/folder1

	[[File:ScratchSpace.png\|center\|frame]]		[[File:ScratchSpace.png\|center\|frame]]

Jacobg2 at 16:59, 20 March 2026

2026-03-20T16:59:33Z

← Older revision		Revision as of 16:59, 20 March 2026
Line 1:		Line 1:
	Every time you read/write from your "/home/yourusername" directory, you are accessing a remote storage location. This means slowdowns can occur, especially if other users are doing the same.		Every time you read/write from your "/home/yourusername" directory, you are accessing a remote storage location. This means slowdowns can occur, especially if other users are doing the same.



Line 7:		Line 8:

	Spinning Disk Storage - Slow machanical hard drive with spinning disks. Larger available storage size		Spinning Disk Storage - Slow machanical hard drive with spinning disks. Larger available storage size



	Each Leviathan is equipped with a 4TB NVME SSD (faster than the NAS). This allows for heavy reading and writing at high speeds. Utilizing this space can speed up your jobs, and help with congestion on the NAS.		Each Leviathan is equipped with a 4TB NVME SSD (faster than the NAS). This allows for heavy reading and writing at high speeds. Utilizing this space can speed up your jobs, and help with congestion on the NAS.

			[[File:ScratchSpace.png\|center\|frame]]

Jacobg2 at 15:11, 20 March 2026

2026-03-20T15:11:07Z

← Older revision		Revision as of 15:11, 20 March 2026
Line 1:		Line 1:
	Every time you read/write from your "/home/yourusername" directory, you are accessing a remote storage location. This means slowdowns can occur, especially if other users are doing the same.		Every time you read/write from your "/home/yourusername" directory, you are accessing a remote storage location. This means slowdowns can occur, especially if other users are doing the same.

	Each Leviathan is equipped with a 4TB SSD (faster than the NAS). This allows for heavy reading and writing at high speeds. Utilizing this space can speed up your jobs, and help with congestion on the NAS.
			'''Terms'''

			NVME SSD - Fast flash based storage. Smaller available storage size.

			Spinning Disk Storage - Slow machanical hard drive with spinning disks. Larger available storage size


			Each Leviathan is equipped with a 4TB NVME SSD (faster than the NAS). This allows for heavy reading and writing at high speeds. Utilizing this space can speed up your jobs, and help with congestion on the NAS.

Jacobg2 at 14:54, 20 March 2026

2026-03-20T14:54:15Z

← Older revision		Revision as of 14:54, 20 March 2026
Line 1:		Line 1:
	Every time you read/~~writre~~ from your /home directory.		Every time you read/write from your "/home/yourusername" directory, you are accessing a remote storage location. This means slowdowns can occur, especially if other users are doing the same.

	Each Leviathan is equipped with a 4TB SSD (faster than the NAS). This allows for heavy reading and writing at high speeds. Utilizing this space can speed up your jobs, and help with congestion on the NAS.		Each Leviathan is equipped with a 4TB SSD (faster than the NAS). This allows for heavy reading and writing at high speeds. Utilizing this space can speed up your jobs, and help with congestion on the NAS.