Scratch Storage: Difference between revisions
No edit summary |
No edit summary |
||
| (9 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
Each Leviathan is equipped with a 4TB SSD (faster than the NAS). This allows for heavy reading and writing at high speeds. Utilizing this space can speed up your jobs | [[Category:Remote Server Access]] | ||
[[Category:Big Data Tools]] | |||
[[Category:Department Servers]] | |||
[[Category:Wright Cluster]] | |||
[[Category:PSC]] | |||
Scratch storage is a location where you can store data directly on the machine. These locations use faster flash based storage, which is better suited for the heavy reading and writing of data. | |||
Every time you read/write from your <code>/home/yourusername</code> directory, you are accessing a remote storage location over the network. This is slower to begin with, especially if other users are running read/write heavy jobs. | |||
== Storage Terminology == | |||
NVME SSD - Fast flash based storage. Smaller available storage size. | |||
Spinning Disk Storage - Slow mechanical hard drive with spinning disks. Larger available storage size | |||
Each Leviathan is equipped with a 4TB NVME SSD (faster than the NAS which uses spinning disks.). This allows for heavy reading and writing at high speeds. Utilizing this space can speed up your jobs and help with congestion on the NAS. | |||
== Use Cases == | |||
Scratch storage is useful for temporarily storing large files you will be reading and writing from. Once your job is completed, you can move the data over to the NAS (/home/yourusername). | |||
Good uses include: | |||
* Saving checkpoints or output from long-running jobs or simulations, to be analyzed and moved to long-term storage once the job is done | |||
* Saving output from one script that will immediately be used by another script | |||
* Writing huge outputs from scripts when you're still debugging them and won't need to keep the output for long | |||
Anything saved to <code>/scratch</code> can be moved (with <code>cp</code> or <code>mv</code>) to your home directory if you decide to keep it permanently. | |||
== Scratch Storage Location on Each Leviathan == | |||
Scratch storage is available in <code>/scratch</code>. | |||
You can view the amount of space used at <code>/scratch</code> for each leviathan [https://info.stat.cmu.edu/grafana/d/asjkldfbnvajdsfh213/leviathan-cluster?orgId=1 here]. | |||
Every <code>/scratch</code> location on each leviathan is it's own storage location. Meaning if you create data on Leviathan1 on <code>/scratch</code>, it will not be accessible on Leviathan7 at <code>/scratch</code>. | |||
====== Steps to use Scratch Storage ====== | |||
# Make a folder in /scratch location. | |||
#* <code>mkdir /scratch/yourusername</code> | |||
# Change ownership of the folder to you. | |||
#* <code>chown yourusername:yourusername /scratch/yourusername</code> | |||
# Update permissions on folder so only you can read it (modify as needed.) | |||
#* <code>chmod 750 /scratch/yourusername</code> | |||
[[File:ScratchSpace.png|center|frame]] | |||
Latest revision as of 03:34, 29 June 2026
Scratch storage is a location where you can store data directly on the machine. These locations use faster flash based storage, which is better suited for the heavy reading and writing of data.
Every time you read/write from your /home/yourusername directory, you are accessing a remote storage location over the network. This is slower to begin with, especially if other users are running read/write heavy jobs.
Storage Terminology
NVME SSD - Fast flash based storage. Smaller available storage size.
Spinning Disk Storage - Slow mechanical hard drive with spinning disks. Larger available storage size
Each Leviathan is equipped with a 4TB NVME SSD (faster than the NAS which uses spinning disks.). This allows for heavy reading and writing at high speeds. Utilizing this space can speed up your jobs and help with congestion on the NAS.
Use Cases
Scratch storage is useful for temporarily storing large files you will be reading and writing from. Once your job is completed, you can move the data over to the NAS (/home/yourusername).
Good uses include:
- Saving checkpoints or output from long-running jobs or simulations, to be analyzed and moved to long-term storage once the job is done
- Saving output from one script that will immediately be used by another script
- Writing huge outputs from scripts when you're still debugging them and won't need to keep the output for long
Anything saved to /scratch can be moved (with cp or mv) to your home directory if you decide to keep it permanently.
Scratch Storage Location on Each Leviathan
Scratch storage is available in /scratch.
You can view the amount of space used at /scratch for each leviathan here.
Every /scratch location on each leviathan is it's own storage location. Meaning if you create data on Leviathan1 on /scratch, it will not be accessible on Leviathan7 at /scratch.
Steps to use Scratch Storage
- Make a folder in /scratch location.
mkdir /scratch/yourusername
- Change ownership of the folder to you.
chown yourusername:yourusername /scratch/yourusername
- Update permissions on folder so only you can read it (modify as needed.)
chmod 750 /scratch/yourusername
