Parallel computing - Revision history

Ajshen at 03:25, 29 June 2026

2026-06-29T03:25:04Z

← Older revision		Revision as of 03:25, 29 June 2026
Line 1:		Line 1:

			[[Category:Parallelization]]

	= When and why of parallel computing =		= When and why of parallel computing =

Skipper at 01:34, 24 May 2024

2024-05-24T01:34:47Z

← Older revision		Revision as of 01:34, 24 May 2024
Line 24:		Line 24:

	= General: When to parallelize (and when not to) =		= General: When to parallelize (and when not to) =
	~~[[File:1200px-Parallelization.png\|600px\|thumb\|center]]~~
	Supposing you’re in a situation where your analysis’s computation is '''cpu-bound''', a secondary question is “can parallelization help speed this up?”. This question should capture		Supposing you’re in a situation where your analysis’s computation is '''cpu-bound''', a secondary question is “can parallelization help speed this up?”. This question should capture
	<ol>		<ol>
Line 37:		Line 36:
	<li>'''Amdahl’s Law''': trade-oof between cores and parallelizable proportion of analysis In the computing community some of these trade-offs are captured by '''Amdahl’s Law''' which relates the increased speedup of your computation (y-axis) as a function of the number of cores (x-axis) and the proportion of computation that can be parallelized (see the colored lines). The image below visualizes this idea:		<li>'''Amdahl’s Law''': trade-oof between cores and parallelizable proportion of analysis In the computing community some of these trade-offs are captured by '''Amdahl’s Law''' which relates the increased speedup of your computation (y-axis) as a function of the number of cores (x-axis) and the proportion of computation that can be parallelized (see the colored lines). The image below visualizes this idea:

	[[File:Parallelization.png\|frameless\|~~600px\|alt=amdahls law~~\|~~amdahls law~~]]		[[File:1200px-Parallelization.png\|600px\|frameless\|thumb\|center]]
	<br>		<br>

Skipper at 01:33, 24 May 2024

2024-05-24T01:33:57Z

← Older revision		Revision as of 01:33, 24 May 2024
Line 24:		Line 24:

	= General: When to parallelize (and when not to) =		= General: When to parallelize (and when not to) =
			[[File:1200px-Parallelization.png\|600px\|thumb\|center]]
	Supposing you’re in a situation where your analysis’s computation is '''cpu-bound''', a secondary question is “can parallelization help speed this up?”. This question should capture		Supposing you’re in a situation where your analysis’s computation is '''cpu-bound''', a secondary question is “can parallelization help speed this up?”. This question should capture
	<ol>		<ol>

Skipper: Created page with "= When and why of parallel computing = Parallel computing is a “hot” topic in computing and data science, but in many Statisticians’ daily lives it’s not too commonpl..."

2024-03-29T23:44:30Z

Created page with "= When and why of parallel computing = Parallel computing is a “hot” topic in computing and data science, but in many Statisticians’ daily lives it’s not too commonpl..."

New page

= When and why of parallel computing =

Parallel computing is a “hot” topic in computing and data science, but in many Statisticians’ daily lives it’s not too commonplace. This wiki article attempts to discuss some examples that might lead you to leveraging parallel computing as a statistician (often in the form of task-parallelization) and the trade-offs and potential power of parallel computing.

= General: Why parallel computing =

Some classic examples (expanded base on a list from [https://people.duke.edu/~ccc14/sta-663-2018/notebooks/S14A_Parallel_Programming_Introduction.html duke] for parallelization in statistics include

# simulations
## sensitivity testing
## parameter tuning
## Monte Carlo integration
# bootstrapping /other resampling
# Multiple MCMC chains

A lot of <code>R</code> or <code>python</code> code runs fast enough on a single processor. But at times, computations can be:

* '''cpu-bound''': Take too much cpu time
* '''memory-bound''': Take too much memory
* '''I/O-bound''': Take too much time to read/write from disk
* '''network-bound''': Take too much time to transfer

Parallelization directly helps with '''cpu-bound''' computations and can allow the user to leverage modern processor architectures that provide multiple cores on a single processor.

= General: When to parallelize (and when not to) =

Supposing you’re in a situation where your analysis’s computation is '''cpu-bound''', a secondary question is “can parallelization help speed this up?”. This question should capture
<ol>
<li>the amount of time you need to parallelize the process & combine the data,</li>
<li>how much increased overhead (in RAM & memory and cpu processing) is required to run and</li>
<li>if the process can even be parallelized.</li>
</ol>

== Potential to parallelize ==

<ol>
<li>'''Amdahl’s Law''': trade-oof between cores and parallelizable proportion of analysis In the computing community some of these trade-offs are captured by '''Amdahl’s Law''' which relates the increased speedup of your computation (y-axis) as a function of the number of cores (x-axis) and the proportion of computation that can be parallelized (see the colored lines). The image below visualizes this idea:

[[File:Parallelization.png|frameless|600px|alt=amdahls law|amdahls law]]
<br>

For example, suppose you have to process a lot of early steps before running something that can be parallelized, and these early steps can’t be saved as they are too memory intensive (e.g. having more RAM than actually solid-state memory available), This might make the amount of computation that can be parallelized smalled (say 50%). In certain cases, this should discourage you to parallelize your process completely. This could lead your to do language-specific parallelization ([[parallelization_in_r|R parallel computing]] or [[parallelization_in_python|Python parallel computing]]) instead of higher-level parallelization (through [[Slurm]] or [[Bash_files|Bash files]] + [[Terminal_multiplexers|Terminal multiplexers]]).
</li>
<li>Memory Constraints

Outside considerations of Amdahl’s Law, the amount of RAM (especially on your personal computer), may give you pause. Often, when trying to alleviate cpu-bound constraints using parallelization one can run into memory-bound constraints. Specifically, running multiple processes intrinsically increases the amount of RAM memory used. this can actually make language-specific parallelization ([[parallelization_in_r|R parallel computing]] or [[parallelization_in_python|Python parallel computing]]) on personal computers '''unworkable''', and encourage utilizing resources with more RAM which can lead to using ([[Slurm]] or [[Bash files]] + [[Terminal_multiplexers]])-style parallelization.
</li></ol>

= Additional Reading =

As you can guess, there are more specific articles to help you get started/know some of what is available to actually do parallelization.

* Language-specific parallelization
** [[Parallelization_in_r|paralleliation with R]]
** [[Parallelization_in_python|parallelization with python]]
* Parallelization on Department servers
** You can use language-specific parallelization or combine [[Terminal multiplexers]] and [[Bash files]] to speed up the process
* [[Pittsburgh Super computer|PSC]]
** you’re best off learning the [[Slurm|Slurm workload manager]]