What resources when

To try to answer the common question “what resources do I have access to and when should I use them”, below we present to approaches to answering the question. The first of which focuses on what resources exist and what they should be used for and the second tries to answer resource needs / current computing problems with resources to fit those needs/ solve those problems.

After you’re found the resource you think you’d like to use (or even when you’re still deciding), we strongly encourage you to attend an computing office hour to help discuss all the things you should be thinking about and get you on the right track quickly.

What computing resources do you have access to

Your personal computer

Your personal computer is great for small jobs that take less than 30 minutes, uses few cores and not too much RAM.

Pro:

1. Naturally the easiest tool to use since it’s yours. Best for short jobs that require only a few cores.

Cons:

capacity:

1. • few cores: 4-8 cores common on laptops<ref>

     From laptopmag

     </ref> (multiply processes per cors means = 8-16 “cores”).
   • low RAM: 4-32 GB of common on laptops<ref>

     From digitaltrends

     </ref>, used across all cores
   • “no” GPUs: you don’t have GPUs that are optimized for Deep Learning

other usage: You often want to use your computer for other things (and it’s built for that exact thing). It’s possible you or someone you know has has had to leave their laptop at home for the day/ run lode oernight When they need to run something intensive -OR- long. But you need that laptop for other work / homework…

The below XKCD comic doesn’t have to be you… (though breaks are good):

https://xkcd.com/303/

Conclusion: Your personal computer isn’t the best for when doing a lot of computing and its easy to put jobs elsewhere!

Common use-cases

1. Most class projects
2. Non-data problems
3. Non-complex structure problems

Associated Articles:

• parallelization

The hydras (the department servers)

The departmental servers should be seen as a “sandbox”. You can experiment with lots of things on the servers and it doesn’t cost you much to do. The hydras have a decent amount of RAM (much more than your personal computer), but often it won’t run a lot faster than if you put you’re computer aside and did the same thing on it (though a large pro is you don’t have to stop using your own computer when running jobs on the hydras).

Pros:

1. Best for long but smaller core jobs
2. Decent amount of RAM per machine (64 GB, or 128 GB on hydra7 and hydra8)
3. Very low barrier to entry (aka easy to get started)
4. File management system shares across different hydra servers

Cons:

1. usage: The department server is a bit of a “wild-west” in terms of usage (it varies how many people use it at a given time)
2. cores per server: the number of cores per server is only 12

Conclusion: Great place to start for more intensive computing.

Associated Articles:

• Linux and department servers

Server Monitoring

View current Hydra resource usage - Hydra Cluster Usage Monitor

PSC (Pittsburgh supercomputer center)

The Pittsburgh supercomputer center provides supercomputers than have massive amounts of cores (but also can find some with large RAMs and GPUs). Generally speaking, the supercomputing resources are often used when you have extremely parallelizable code that would take a very long time if you ran it on just a few cores (e.g. your computer or a few hydras).

Pros:

1. A massive amount of cores and RAM that can be leveraged for extremely parallelizable computing
2. Easier to access in the statistics department

Cons:

1. Requires learning of new job management system (though quick to learn - see [Slurm])
2. Some options to access the resources can take time or require going through your advisor
3. currently (2021) has been having problems with servers crashing

Associated Articles:

• PSC
• Slurm

GPU Computing

The Statistics & Data Science Department offers two main GPU computing environments.

Ghidorahs

• 3 physical servers
• RTX 4090 GPUs
• Connected to the Stats network share
• Your statistics directory account will grant you access

Wright

• For a detail overview, please check here
• You must request access
• The Statistics network share is not accessible

Computing resource flowchart

These flow charts try to help capture what’s available and when you might use them.

Visit the links below to view current computing resource usage. Select a server with the least amount of users and lowest resource usage.

• Hydra Cluster Usage Monitor

• Ghidorah Cluster Usage Monitor

From laptopmag
     2. ↑ From digitaltrends