Parallelization: Making the most of your Machine: Difference between revisions
From Stat DS Computing Wiki
No edit summary |
No edit summary |
||
| Line 9: | Line 9: | ||
!Summary | !Summary | ||
!Technical Level | !Technical Level | ||
! | !Relevance | ||
|- | |- | ||
|[[Parallel computing]] | |[[Parallel computing]] | ||
|A primer on the benefits of parallel computing | |A primer on the benefits of parallel computing | ||
|⭐⚫⚫⚫⚫ | |⭐⚫⚫⚫⚫ | ||
| | | style="background-color: #44ee88;" |Almost all computing tasks | ||
|- | |- | ||
|[[Terminal multiplexers]] | |[[Terminal multiplexers]] | ||
|Use multiple terminal "windows" to run several workflows simultaneously | |Use multiple terminal "windows" to run several workflows simultaneously | ||
|⭐⭐⚫⚫⚫ | |⭐⭐⚫⚫⚫ | ||
| | |Small scale parallelization | ||
|- | |- | ||
|[[Parallelization in python]] | |[[Parallelization in python]] | ||
|Introduces several python packages for parallel workflows | |Introduces several python packages for parallel workflows | ||
|⭐⭐⭐⚫⚫ | |⭐⭐⭐⚫⚫ | ||
| | |Python workflows only | ||
|- | |- | ||
|[[Parallelization in r]] | |[[Parallelization in r]] | ||
|Introduces several R packages for parallel workflows | |Introduces several R packages for parallel workflows | ||
|⭐⭐⭐⚫⚫ | |⭐⭐⭐⚫⚫ | ||
| | |R workflows only | ||
|- | |- | ||
|[[Slurm#Using SLURM to parallelize workflows|Using SLURM to parallelize workflows]] | |[[Slurm#Using SLURM to parallelize workflows|Using SLURM to parallelize workflows]] | ||
|How to use the SLURM job scheduler to parallelize workflows | |How to use the SLURM job scheduler to parallelize workflows | ||
|⭐⭐⭐⭐⚫ | |⭐⭐⭐⭐⚫ | ||
| | |HPC workflows with SLURM scheduling | ||
|} | |} | ||
Revision as of 05:32, 27 March 2026
By default, most programming languages like Python and R will only use one CPU core/thread to execute your code. For small workloads, this is often sufficient even on a personal device. However, larger data processing or analysis workflows will often taken far too long to run on a single core or thread.
Parallelization is the process of taking a large computational task and breaking it into smaller tasks that can all be run simultaneously and independently. For example, most bootstrap methods can be parallelized, since each bootstrap sample is processed independently. This page will introduce several tools and techniques for running parallel workflows.
Back to Beginner's Guide to Computation in Statistical Research
| Title | Summary | Technical Level | Relevance |
|---|---|---|---|
| Parallel computing | A primer on the benefits of parallel computing | ⭐⚫⚫⚫⚫ | Almost all computing tasks |
| Terminal multiplexers | Use multiple terminal "windows" to run several workflows simultaneously | ⭐⭐⚫⚫⚫ | Small scale parallelization |
| Parallelization in python | Introduces several python packages for parallel workflows | ⭐⭐⭐⚫⚫ | Python workflows only |
| Parallelization in r | Introduces several R packages for parallel workflows | ⭐⭐⭐⚫⚫ | R workflows only |
| Using SLURM to parallelize workflows | How to use the SLURM job scheduler to parallelize workflows | ⭐⭐⭐⭐⚫ | HPC workflows with SLURM scheduling |
