Parallel computing

When and why of parallel computing

Parallel computing is a “hot” topic in computing and data science, but in many Statisticians’ daily lives it’s not too commonplace. This wiki article attempts to discuss some examples that might lead you to leveraging parallel computing as a statistician (often in the form of task-parallelization) and the trade-offs and potential power of parallel computing.

General: Why parallel computing

Some classic examples (expanded base on a list from duke for parallelization in statistics include

1. simulations
   1. sensitivity testing
   2. parameter tuning
   3. Monte Carlo integration
2. bootstrapping /other resampling
3. Multiple MCMC chains

A lot of R or python code runs fast enough on a single processor. But at times, computations can be:

• cpu-bound: Take too much cpu time
• memory-bound: Take too much memory
• I/O-bound: Take too much time to read/write from disk
• network-bound: Take too much time to transfer

Parallelization directly helps with cpu-bound computations and can allow the user to leverage modern processor architectures that provide multiple cores on a single processor.

General: When to parallelize (and when not to)

Supposing you’re in a situation where your analysis’s computation is cpu-bound, a secondary question is “can parallelization help speed this up?”. This question should capture

1. the amount of time you need to parallelize the process & combine the data,
2. how much increased overhead (in RAM & memory and cpu processing) is required to run and
3. if the process can even be parallelized.

Potential to parallelize

1. Amdahl’s Law: trade-oof between cores and parallelizable proportion of analysis In the computing community some of these trade-offs are captured by Amdahl’s Law which relates the increased speedup of your computation (y-axis) as a function of the number of cores (x-axis) and the proportion of computation that can be parallelized (see the colored lines). The image below visualizes this idea:

   

   thumb

   
   

   For example, suppose you have to process a lot of early steps before running something that can be parallelized, and these early steps can’t be saved as they are too memory intensive (e.g. having more RAM than actually solid-state memory available), This might make the amount of computation that can be parallelized smalled (say 50%). In certain cases, this should discourage you to parallelize your process completely. This could lead your to do language-specific parallelization (R parallel computing or Python parallel computing) instead of higher-level parallelization (through Slurm or Bash files + Terminal multiplexers).

2. Memory Constraints Outside considerations of Amdahl’s Law, the amount of RAM (especially on your personal computer), may give you pause. Often, when trying to alleviate cpu-bound constraints using parallelization one can run into memory-bound constraints. Specifically, running multiple processes intrinsically increases the amount of RAM memory used. this can actually make language-specific parallelization (R parallel computing or Python parallel computing) on personal computers unworkable, and encourage utilizing resources with more RAM which can lead to using (Slurm or Bash files + Terminal_multiplexers)-style parallelization.

Additional Reading

As you can guess, there are more specific articles to help you get started/know some of what is available to actually do parallelization.

• Language-specific parallelization
  • paralleliation with R
  • parallelization with python
• Parallelization on Department servers
  • You can use language-specific parallelization or combine Terminal multiplexers and Bash files to speed up the process
• PSC
  • you’re best off learning the Slurm workload manager