Parallelization in python - Revision history

Ajshen at 02:34, 30 June 2026

2026-06-30T02:34:15Z

← Older revision		Revision as of 02:34, 30 June 2026
Line 1:		Line 1:

			[[Category:Parallelization]]

	In Python, parallelization is a bit more of a “wild west” in terms of the number of parallelization packages available. This wiki will attempt to present minor examples of some common approaches, along with an outline of things to think about in selecting the library you end up using. In general, the process in Python involves:		In Python, parallelization is a bit more of a “wild west” in terms of the number of parallelization packages available. This wiki will attempt to present minor examples of some common approaches, along with an outline of things to think about in selecting the library you end up using. In general, the process in Python involves:

Line 23:		Line 26:
	def my_simulation(k):		def my_simulation(k):
	"""		"""
	~~estamine~~ the polynomial fit of degree k's MSE for a linear		estimate the polynomial fit of degree k's MSE for a linear
	relationship		relationship

Line 136:		Line 139:
	results = ray.get(futures)		results = ray.get(futures)
	print(str(time.time() - starttime) + " seconds")		print(str(time.time() - starttime) + " seconds")
	</syntaxhighlight>This <code>.remote()</code> piece is useful because you can pass previous computations into it in order to have a parallelized, autoregressive work flow.<syntaxhighlight>		</syntaxhighlight>This <code>.remote()</code> piece is useful because you can pass previous computations into it in order to have a parallelized, autoregressive work flow.<syntaxhighlight lang="python">
	0.13494515419006348 seconds		0.13494515419006348 seconds
	</syntaxhighlight>		</syntaxhighlight>

Ecucuzze at 14:01, 1 July 2025

2025-07-01T14:01:29Z

← Older revision		Revision as of 14:01, 1 July 2025
Line 1:		Line 1:
	~~'''This article is still being developed - released earlier to align with Department seminar.'''~~

	In Python, parallelization is a bit more of a “wild west” in terms of the number of parallelization packages available. This wiki will attempt to present minor examples of some common approaches, along with an outline of things to think about in selecting the library you end up using. In general, the process in Python involves:		In Python, parallelization is a bit more of a “wild west” in terms of the number of parallelization packages available. This wiki will attempt to present minor examples of some common approaches, along with an outline of things to think about in selecting the library you end up using. In general, the process in Python involves:

Line 209:		Line 207:
	= Additional Topics and Resources =		= Additional Topics and Resources =

	~~# [https://ray.io/ <code>Ray</code> python package] allows paralleilization across multiple computers~~		# [https://people.duke.edu/~ccc14/sta-663-2018/notebooks/S14A_Parallel_Programming_Introduction.html Duke’s statistical computing course] provides a set of examples for multiple Python parallelization packages
	# [https://people.duke.edu/~ccc14/sta-663-2018/notebooks/S14A_Parallel_Programming_Introduction.html Duke’s statistical computing course] provides a set of examples for multiple ~~python~~ parallelization packages \|

Ecucuzze: /* Just in Time compilers + parallelization */

2025-06-30T18:30:57Z

Just in Time compilers + parallelization

← Older revision		Revision as of 18:30, 30 June 2025
Line 129:		Line 129:
	import os</syntaxhighlight>In order to parallelize computation with <code>ray</code>, you must first initialize an instance of <code>ray</code>. Just like with <code>multiprocess</code>, the instance relies on the number of CPUs available.<syntaxhighlight lang="python">		import os</syntaxhighlight>In order to parallelize computation with <code>ray</code>, you must first initialize an instance of <code>ray</code>. Just like with <code>multiprocess</code>, the instance relies on the number of CPUs available.<syntaxhighlight lang="python">
	ray.init(num_cpus = os.cpu_count())		ray.init(num_cpus = os.cpu_count())
	</syntaxhighlight>Additionally, in order for <code>ray</code> to work, the function <code>my_simulation</code> needs to have the ~~docker~~ <code>@ray.remote</code> prior to its heading. <syntaxhighlight lang="python">		</syntaxhighlight>Additionally, in order for <code>ray</code> to work, the function <code>my_simulation</code> needs to have the decorator <code>@ray.remote</code> prior to its heading. <syntaxhighlight lang="python">
	@ray.remote		@ray.remote
	def my_simulation(k):		def my_simulation(k):
	"""the rest of the function code goes here"""		"""the rest of the function code goes here"""
	</syntaxhighlight>Once we have initialized an instance based on our number of available CPUs, and we have made sure our function has the proper ~~docker~~, we can parallelize computation as follows. The <code>.remote()</code> passes the variable into the function, even if the previous computation has not finished running, and the <code>.get()</code> compiles the results once finished.<syntaxhighlight lang="python">		</syntaxhighlight>Once we have initialized an instance based on our number of available CPUs, and we have made sure our function has the proper decorator, we can parallelize computation as follows. The <code>.remote()</code> passes the variable into the function, even if the previous computation has not finished running, and the <code>.get()</code> compiles the results once finished.<syntaxhighlight lang="python">
	starttime = time.time()		starttime = time.time()
	futures = [my_simulation.remote(k) for k in my_k]		futures = [my_simulation.remote(k) for k in my_k]
Line 186:		Line 186:
	results = dv.map_async(my_simulation, my_k).get() </syntaxhighlight>		results = dv.map_async(my_simulation, my_k).get() </syntaxhighlight>
	<syntaxhighlight lang="python">209 ms ± 28.1 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)</syntaxhighlight>		<syntaxhighlight lang="python">209 ms ± 28.1 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)</syntaxhighlight>
	= Just in Time ~~compilers~~ + parallelization =		= Just in Time Compilers + Parallelization =

			Just-in-time compilation is the compiling of your computer code during execution instead of beforehand. This means bytecode is not translated to machine code until run time, and is then executed directly. Code is continuously analyzed to determine where compilation/recompilation would speed up performance. Used in conjunction with parallelization, this is an efficient way to improve computational speed.

			== <code>numba</code> ==
			<code>numba</code> is a high performance Python compiler that translates Python/numpy code into fast, efficient machine code. Python is traditionally regarded as a "slow" language; using <code>numba</code>, you can approach the speeds of languages such as C. <code>numba</code> is specifically designed to work well with numpy arrays.

			=== Implementation ===
			Similar to <code>ray</code>, <code>numba</code> requires a decorator <code>@njit (parallel = True)</code> before the function. We will define a different function to be parallelized here to demonstrate <code>numba</code> <code>prange</code> which allows for efficient parallelization across loops.<syntaxhighlight lang="python">
			A = np.random.rand(3, 3)
			B = np.random.rand(3, 3)

			@njit(parallel = True)
			def my_simulation(A, B):
			C = np.empty_like(A)
			for i in prange(A.shape[0]):
			for j in range(A.shape[1]):
			C[i, j] = A[i, j] + B[i, j]
			return C
			</syntaxhighlight><code>prange</code> should not be nested and should only ever be used in the outer loop. This is a very simple example, but for computationally complex looping problems, <code>numba</code> is highly efficient. However, you need to ensure that you are not creating or declaring a variable within a loop that cannot be accessed outside of the loop, or else parallelization will not work.

	TODO: <code>numba</code> and <code>cython</code>. The idea of these is to compile the function you will be using the first time you use it and then never compile it again (speeding up the actual use of the function). That is then combined with standard parallelization
	= Additional Topics and Resources =		= Additional Topics and Resources =

	# [https://ray.io/ <code>Ray</code> python package] allows paralleilization across multiple computers		# [https://ray.io/ <code>Ray</code> python package] allows paralleilization across multiple computers
	# [https://people.duke.edu/~ccc14/sta-663-2018/notebooks/S14A_Parallel_Programming_Introduction.html Duke’s statistical computing course] provides a set of examples for multiple python parallelization packages \|		# [https://people.duke.edu/~ccc14/sta-663-2018/notebooks/S14A_Parallel_Programming_Introduction.html Duke’s statistical computing course] provides a set of examples for multiple python parallelization packages \|

Ecucuzze: Added a section on ray

2025-06-25T17:29:21Z

Added a section on ray

@@ Line 121: / Line 121: @@
 <syntaxhighlight lang="python">r = Parallel(n_jobs=num_cores-2, verbose=10)(delayed(my_simulation)(k) for k in my_k)</syntaxhighlight></li>
 <li><p>'''Multiple parameters''' can be passed with redefinitions of functions to take groupings. To replace multiple for-loops, often the function <code>itertools.product()</code> is used for the parameters.</p></li></ol>
 = Jupyter Notebook: <code>ipyparallel</code>=

Ecucuzze: Updated the joblibs times to match what I am currently getting

2025-06-25T16:31:26Z

Updated the joblibs times to match what I am currently getting

← Older revision		Revision as of 16:31, 25 June 2025
Line 83:		Line 83:
	==== Synchronous ====		==== Synchronous ====

	<syntaxhighlight lang="python">~~%%timeit~~		<syntaxhighlight lang="python">starttime = time.time()

	with mp.Pool(processes=num_cores-2) as p:		with mp.Pool(processes=num_cores-2) as p:
	results = p.map(my_simulation, my_k)</syntaxhighlight>		results = p.map(my_simulation, my_k)
	<syntaxhighlight lang="python">~~209 ms ± 24~~.~~7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)~~</syntaxhighlight>		print(str(time.time() - starttime) + " seconds")</syntaxhighlight>
			<syntaxhighlight lang="python">0.7638578414916992 seconds</syntaxhighlight>
	==== Asynchronous ====		==== Asynchronous ====

	<syntaxhighlight lang="python">~~%%timeit~~		<syntaxhighlight lang="python">starttime = time.time()

	with mp.Pool(processes=num_cores-2) as p:		with mp.Pool(processes=num_cores-2) as p:
	results = p.map_async(my_simulation, my_k).get() ~~# not clear why I needed to use~~ .~~get~~() ~~to get something returnable for async~~</syntaxhighlight>		results = p.map_async(my_simulation, my_k).get()
	<syntaxhighlight lang="python">~~48.9 ms ± 3.08 ms per loop (mean ± std~~. ~~dev. of 7 runs, 10 loops each)~~</syntaxhighlight>		print(str(time.time() - starttime) + " seconds")
			</syntaxhighlight>
			<syntaxhighlight lang="python">0.8183681964874268 seconds</syntaxhighlight>
	== <code>joblib</code> ==		== <code>joblib</code> ==

Line 108:		Line 109:
	We then can move our list-recursive example into our <code>joblib</code> structure. We wrap <code>Parallel(...)()</code> around the inside of the list-recursion and <code>delayed()</code> around the functions that will be repeated multiple times.		We then can move our list-recursive example into our <code>joblib</code> structure. We wrap <code>Parallel(...)()</code> around the inside of the list-recursion and <code>delayed()</code> around the functions that will be repeated multiple times.

	<syntaxhighlight lang="python">~~%% timeit~~		<syntaxhighlight lang="python">starttime = time.time()
			r = Parallel(n_jobs=num_cores-2)(delayed(my_simulation)(k) for k in my_k)
	r = Parallel(n_jobs=num_cores-2)(delayed(my_simulation)(k) for k in my_k)</syntaxhighlight>		print(str(time.time() - starttime) + " seconds")</syntaxhighlight>
	<syntaxhighlight lang="python">~~172 ms ± 23~~.~~4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)~~</syntaxhighlight>		<syntaxhighlight lang="python">0.5306367874145508 seconds</syntaxhighlight>
	=== Add-ons ===		=== Add-ons ===

Ecucuzze: Updated with the new times I'm getting so the results are consistent

2025-06-25T16:24:59Z

Updated with the new times I'm getting so the results are consistent

← Older revision		Revision as of 16:24, 25 June 2025
Line 57:		Line 57:
	return(np.mean((y0-y0_hat)**2))		return(np.mean((y0-y0_hat)**2))
	</syntaxhighlight>		</syntaxhighlight>
	<syntaxhighlight lang="python">~~%%timeit -n3~~		<syntaxhighlight lang="python">import time

	[my_simulation(k) for k in my_k]</syntaxhighlight>		starttime = time.time()
	<syntaxhighlight lang="python">~~676 ms ± 37~~.~~6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)~~</syntaxhighlight>		[my_simulation(k) for k in my_k]
			print(str(time.time() - starttime) + " seconds")</syntaxhighlight>
			<syntaxhighlight lang="python">0.9454693794250488 seconds</syntaxhighlight>
	= General Python Parallelization Packages =		= General Python Parallelization Packages =

Ecucuzze at 16:00, 25 June 2025

2025-06-25T16:00:00Z

← Older revision		Revision as of 16:00, 25 June 2025
Line 119:		Line 119:
	<li><p>'''Multiple parameters''' can be passed with redefinitions of functions to take groupings. To replace multiple for-loops, often the function <code>itertools.product()</code> is used for the parameters.</p></li></ol>		<li><p>'''Multiple parameters''' can be passed with redefinitions of functions to take groupings. To replace multiple for-loops, often the function <code>itertools.product()</code> is used for the parameters.</p></li></ol>

	= ~~Just in Time compilers + parallelization~~ =		= Jupyter Notebook: <code>ipyparallel</code>=

	~~TODO:~~ <code>numba</code> and <code>cython</code>. The idea of these is to compile the function you will be using the first time you use it and then never compile it again (speeding up the actual use of the function). That is then combined with standard parallelization		For those that heavily use Jupyter notebooks, the package <code>ipyparallel</code> works quite nicely. Some benefits of this package include its verbosity, its ability to serialize numpy arrays cleanly, and the fact that it can distribute across multiple machines. This package also works with <code>numba.jit</code> and <code>cython</code>.

	~~= Jupyter notebook: <code>ipyparallel</code> =~~		Before opening your <code>jupyter notebook</code> you’ll want to make sure the extension is enabled - which can be done from the command line with the following command or inside the <code>jupyter notebook</code> extensions tab.

	For those that heavily use jupyter notebooks and nothing else (which this author personally doesn’t encourage), the package <code>ipyparallel</code> nicely works inside jupyter notebooks (but so does many of the other packages above like <code>joblib</code>, <code>multiprocess</code>, etc.). One nice this about this package is the verbosity, etc. works intellegently in the notebook. This package also works with <code>numba.jit</code> and <code>cython</code>.

	Before opening your <code>jupyter notebook</code> you’ll want to make sure the extension is enabled - which can be ~~down~~ from the command line with the following command or inside the <code>jupyter notebook</code> extensions tab.

	<syntaxhighlight lang="bash">ipcluster nbextension enable</syntaxhighlight>		<syntaxhighlight lang="bash">ipcluster nbextension enable</syntaxhighlight>
	~~And~~ then ~~you’ll~~ need to go to the “IPython Clusters” tab in <code>jupyter noteook</code> and start some parallel clusters.		You’ll then need to go to the “IPython Clusters” tab in <code>jupyter noteook</code> and start some parallel clusters.

	== <code>ipyparallel</code> parallelization ==		==<code>ipyparallel</code> parallelization ==

	<syntaxhighlight lang="python">import ipyparallel as ipp</syntaxhighlight>		<syntaxhighlight lang="python">import ipyparallel as ipp</syntaxhighlight>
	For direct execution we define <code>dv</code> and provide it with what it ~~needs…~~		For direct execution we define <code>dv</code> and provide it with what it needs.

	<syntaxhighlight lang="python">rc = ipp.Client()		<syntaxhighlight lang="python">rc = ipp.Client()
	dv = rc[:] # direct view</syntaxhighlight>		dv = rc[:] # direct view</syntaxhighlight>
	=== ~~passing things~~ to ~~each cluster~~ ===		=== Passing Things to Each Cluster ===

	<code>ipyparallel</code> provides ways for the user to pass only the necessary items to the the clusters (as to not overload ~~the clusters with too many things)~~.		<code>ipyparallel</code> provides ways for the user to pass only the necessary items to the the clusters as to not overload them.

	<ol style="list-style-type: decimal;">		<ol style="list-style-type: decimal;">
	<li><p>~~passing~~ packages all together</p>		<li><p>Passing packages all together</p>
	<syntaxhighlight lang="python">with dv.sync_imports():		<syntaxhighlight lang="python">with dv.sync_imports():
	from sklearn.linear_model import LinearRegression		from sklearn.linear_model import LinearRegression
	# getting dependencies on the clusters</syntaxhighlight></li>		# getting dependencies on the clusters</syntaxhighlight></li>
	<li><p>~~passing variable over~~:</p>		<li><p>Passing variables:</p>
	<p>Below is just an example - we don’t need to pass <code>my_k</code> since <code>dv</code> will directly get it when we ~~wnat~~ to use it.</p>		<p>Below is just an example - we don’t need to pass <code>my_k</code> since <code>dv</code> will directly get it when we want to use it.</p>
	<syntaxhighlight lang="python">dv["my_k"] = my_k</syntaxhighlight></li>		<syntaxhighlight lang="python">dv["my_k"] = my_k</syntaxhighlight></li>
	<li><p>we can also use the <code>%px</code> magick to run lines of code on all servers~~. Given I think <code>numpy</code> is cooler as <code>np</code> I actually have to do it this way, not the <code>dv.sync_imports</code> way~~.</p>		<li><p>We can also use the <code>%px</code> magick to run lines of code on all servers.</p>
	<syntaxhighlight lang="python">%px import numpy as np</syntaxhighlight></li></ol>		<syntaxhighlight lang="python">%px import numpy as np</syntaxhighlight></li></ol>

	=== ~~actual functionality~~ ===		=== Actual Functionality ===

	The actual functionality is very similar to <code>multiprocess</code>/<code>multiprocessing</code> with <code>maps</code> and <code>apply</code>.		The actual functionality is very similar to <code>multiprocess</code>/<code>multiprocessing</code> with <code>maps</code> and <code>apply</code>.

	==== ~~synchronous~~ ====		==== Synchronous ====

	<syntaxhighlight lang="python">%%timeit		<syntaxhighlight lang="python">%%timeit
	results = dv.map_sync(my_simulation, my_k)</syntaxhighlight>		results = dv.map_sync(my_simulation, my_k)</syntaxhighlight>
	<syntaxhighlight lang="python">201 ms ± 34.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)</syntaxhighlight>		<syntaxhighlight lang="python">201 ms ± 34.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)</syntaxhighlight>
	==== asynchronous ====		==== Asynchronous ====
			For asynchronous, you need to use <code>.get()</code> in order to get the results at a later time/when they come in, since the work is not done in order and is instead chunked for efficiency.<syntaxhighlight lang="python">%%timeit
			results = dv.map_async(my_simulation, my_k).get() </syntaxhighlight>
			<syntaxhighlight lang="python">209 ms ± 28.1 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)</syntaxhighlight>
			= Just in Time compilers + parallelization =

	<~~syntaxhighlight lang="python"~~>~~%%timeit~~		TODO: <code>numba</code> and <code>cython</code>. The idea of these is to compile the function you will be using the first time you use it and then never compile it again (speeding up the actual use of the function). That is then combined with standard parallelization
	~~results = dv.map_async(my_simulation, my_k).get() # not clear why I needed to use .get() to get something returnable for async~~</~~syntaxhighlight~~>		= Additional Topics and Resources =
	<~~syntaxhighlight lang="python"~~>~~209 ms ± 28~~.~~1 ms per loop~~ (~~mean ± std. dev.~~ of ~~7 runs, 10 loops each~~)~~</syntaxhighlight>~~
	= ~~additional topics~~ and ~~resources~~ =

	# [https://ray.io/ <code>Ray</code> python package] allows paralleilization across multiple computers		# [https://ray.io/ <code>Ray</code> python package] allows paralleilization across multiple computers
	# [https://people.duke.edu/~ccc14/sta-663-2018/notebooks/S14A_Parallel_Programming_Introduction.html Duke’s statistical computing course] provides a set of examples for multiple python parallelization packages \|		# [https://people.duke.edu/~ccc14/sta-663-2018/notebooks/S14A_Parallel_Programming_Introduction.html Duke’s statistical computing course] provides a set of examples for multiple python parallelization packages \|

Ecucuzze: Cleaned up and added further explanation to multiprocessing and joblib

2025-06-25T14:49:05Z

Cleaned up and added further explanation to multiprocessing and joblib

← Older revision		Revision as of 14:49, 25 June 2025
Line 13:		Line 13:
	We will also discuss Python packages that are built specifically to parallelize jobs inside '''Jupyter notebooks'''.		We will also discuss Python packages that are built specifically to parallelize jobs inside '''Jupyter notebooks'''.

	= Non-parallelized: ~~single core example~~ =		= Non-parallelized: Single Core Example =

	We naturally want to compare a parallelization approach with the base single core analysis, which can be done with the code as follows:		We naturally want to compare a parallelization approach with the base single core analysis, which can be done with the code as follows:
Line 61:		Line 61:
	[my_simulation(k) for k in my_k]</syntaxhighlight>		[my_simulation(k) for k in my_k]</syntaxhighlight>
	<syntaxhighlight lang="python">676 ms ± 37.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)</syntaxhighlight>		<syntaxhighlight lang="python">676 ms ± 37.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)</syntaxhighlight>
	= General ~~python parallelization packages~~ =		= General Python Parallelization Packages =

	As mentioned there are a lot of ~~package to do~~ general parallelization. Two commonly used ones are <code>joblib</code> and <code>multiprocessing</code>. <code>joblib</code> looks a bit more like list-recursion and <code>multiprocessing</code> uses functions like <code>.map()</code>. <code>multiprocessing</code> can have problems with <code>ipython</code> and <code>jupyter notebook</code>s (though <code>joblib</code> wraps it and <code>multiprocess</code> is just a different fork of <code>multiprocessing</code>).		As previously mentioned, there are a lot of packages for general parallelization. Two commonly used ones are <code>joblib</code> and <code>multiprocessing</code>. <code>joblib</code> looks a bit more like list-recursion and <code>multiprocessing</code> uses functions like <code>.map()</code>.

	== <code>multiprocessing</code>/<code>multiprocess</code> ==		== <code>multiprocessing</code>/<code>multiprocess</code> ==

	The <code>multiprocessing</code> package is actually built into ~~python~~, and works in a pretty straightforward manner. It also allows for synchronous vs asynchronous parallelization, where synchronous means the order of the jobs must be followed and asynchronous ~~where the order of jobs need not~~ be ~~followed~~. ~~Many~~ statistical uses of parallelization ~~does~~ not need synchronous parallelization.		The <code>multiprocessing</code> package is actually built into Python, and works in a pretty straightforward manner. It also allows for synchronous vs. asynchronous parallelization, where synchronous means the order of the jobs must be followed, one after the next, and asynchronous means multiple tasks can be executed in small parts in overlapping periods. Note that many statistical uses of parallelization do not need synchronous parallelization.

	~~There since~~ both approaches have the same syntax we’ll be using <code>multiprocess</code> (as it works with <code>ipython</code> and <code>jupyter notebooks</code>).		Since both approaches have the same syntax we’ll be using <code>multiprocess</code> (as it works with <code>ipython</code> and <code>jupyter notebooks</code>). <code>multiprocessing</code> can have problems with <code>ipython</code> and <code>jupyter notebook</code>s. In <code>jupyter notebook</code>s, if you want to use <code>multiprocessing</code> , it needs to be wrapped in <code>if name == "main"</code> to properly fork processes; you can also save your process is as a .py file and import it into your <code>jupyter notebook.</code>

	=== ~~parallelization~~ ===		=== Implementation ===

	First, you need to identify the number of cores you have:		First, you need to identify the number of cores you have:

	<syntaxhighlight lang="python">import multiprocess as mp		<syntaxhighlight lang="python">import multiprocess as mp
	#import multiprocessing as mp		#Could also using import multiprocessing as mp; the syntax is the same

	num_cores = mp.cpu_count()</syntaxhighlight>		num_cores = mp.cpu_count()</syntaxhighlight>
	==== ~~synchronous~~ ====		==== Synchronous ====

	<syntaxhighlight lang="python">%%timeit		<syntaxhighlight lang="python">%%timeit
Line 86:		Line 86:
	results = p.map(my_simulation, my_k)</syntaxhighlight>		results = p.map(my_simulation, my_k)</syntaxhighlight>
	<syntaxhighlight lang="python">209 ms ± 24.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)</syntaxhighlight>		<syntaxhighlight lang="python">209 ms ± 24.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)</syntaxhighlight>
	==== ~~asynchronous~~ ====		==== Asynchronous ====

	<syntaxhighlight lang="python">%%timeit		<syntaxhighlight lang="python">%%timeit
Line 95:		Line 95:
	== <code>joblib</code> ==		== <code>joblib</code> ==

	The <code>joblib</code> package is pretty straightforward and requires you to structure things in terms of list-recursion with a few wrappers to help the process work as desired.		The <code>joblib</code> package is pretty straightforward and requires you to structure things in terms of list-recursion with a few wrappers to help the process work as desired. Just like with <code>multiprocessing</code>, the number of cores needs to be computed before proceeding.

	=== ~~setup~~ ===		=== Implementation ===

	<syntaxhighlight lang="python">import joblib		<syntaxhighlight lang="python">import joblib

	num_cores = joblib.cpu_count()</syntaxhighlight>		num_cores = joblib.cpu_count()</syntaxhighlight>To use <code>joblib</code> functionality, we can import the functions <code>Parallel</code> and <code>delayed</code>. <code>Parallel</code> creates an instance for parallelization with a set number of jobs based on the core availability. <code>delayed</code> creates deferred function calls to distribute computations across multiple cores; this is the function that allows for simultaneous task execution.
	~~=== implimentation ===~~

	To use <code>joblib</code> functionality we can import the functions <code>Parallel</code> and <code>delayed</code>.

	<syntaxhighlight lang="python">from joblib import Parallel, delayed</syntaxhighlight>		<syntaxhighlight lang="python">from joblib import Parallel, delayed</syntaxhighlight>
	We then can move our list-recursive example into our <code>joblib</code> structure. We wrap <code>Parallel(...)()</code> around ~~the old of~~ the inside of the list-recursion and <code>delayed()</code> around the functions that will be repeated multiple times.		We then can move our list-recursive example into our <code>joblib</code> structure. We wrap <code>Parallel(...)()</code> around the inside of the list-recursion and <code>delayed()</code> around the functions that will be repeated multiple times.

	<syntaxhighlight lang="python">%% timeit		<syntaxhighlight lang="python">%% timeit
Line 118:		Line 115:

	<ol style="list-style-type: decimal;">		<ol style="list-style-type: decimal;">
	<li><p>'''~~verbosity~~''' can be added with a single parameter use (as seen below) ~~- the large the value, the~~ more ~~messages:~~</p>		<li><p>'''Verbosity''' can be added with a single parameter use (as seen below). This will provide more detailed output information.</p>
	<syntaxhighlight lang="python">r = Parallel(n_jobs=num_cores-2, verbose=10)(delayed(my_simulation)(k) for k in my_k)</syntaxhighlight></li>		<syntaxhighlight lang="python">r = Parallel(n_jobs=num_cores-2, verbose=10)(delayed(my_simulation)(k) for k in my_k)</syntaxhighlight></li>
	<li><p>'''~~multiple~~ parameters''' can be passed with redefinitions of functions to take groupings. To replace multiple for-loops often the function <code>itertools.product()</code> is used for the parameters.</p></li></ol>		<li><p>'''Multiple parameters''' can be passed with redefinitions of functions to take groupings. To replace multiple for-loops, often the function <code>itertools.product()</code> is used for the parameters.</p></li></ol>

	= Just in Time compilers + parallelization =		= Just in Time compilers + parallelization =

Ecucuzze at 14:30, 25 June 2025

2025-06-25T14:30:32Z

← Older revision		Revision as of 14:30, 25 June 2025
Line 1:		Line 1:
	'''This article is still being developed - released earlier to align with Department seminar.'''		'''This article is still being developed - released earlier to align with Department seminar.'''

	In ~~python~~, parallelization is a bit more of a “wild west” in terms of the number of packages ~~that can help you parallelize your processes (~~This wiki will attempt to present minor examples of some along with an outline of things to think about in selecting the library you end up using. In general, the process in ~~python~~ involves		In Python, parallelization is a bit more of a “wild west” in terms of the number of parallelization packages available. This wiki will attempt to present minor examples of some common approaches, along with an outline of things to think about in selecting the library you end up using. In general, the process in Python involves:

	# defining a function that captures what you want to have parallelized		# defining a function that captures what you want to have parallelized
	# ~~call~~ that function repeatedly in the the parallelization package way (which often looks like a list-recursive call).		# calling that function repeatedly in the the parallelization package way (which often looks like a list-recursive call).

	There are a few different general frameworks to accomplish this.		There are a few different general frameworks to accomplish this.

	# '''~~general~~ wrappers''', define the cores you’ll be using and then wrap that around the function call in a list-recursive way		# '''General wrappers''' define the cores you’ll be using and then wrap that around the function call in a list-recursive way.
	# '''~~just~~ in time compilers''', in addition to providing parallelization tools ~~these packages~~ provide decorators (e.g. <code>@jit</code>) for functions that allow for just in time compilation of the ~~python~~ function into lower level machine code (like <code>C++</code>)		# '''Just in time compilers''', in addition to providing parallelization tools, provide decorators (e.g. <code>@jit</code>) for functions that allow for just in time compilation of the Python function into lower level machine code (like <code>C++</code>).

	We will discuss ~~a python package~~ that ~~is build~~ specifically to parallelize jobs inside '''~~jupyter~~ notebooks'''.		We will also discuss Python packages that are built specifically to parallelize jobs inside '''Jupyter notebooks'''.

	= Non-parallelized: single core example =		= Non-parallelized: single core example =

Ecucuzze at 15:35, 24 June 2025

2025-06-24T15:35:40Z

← Older revision		Revision as of 15:35, 24 June 2025
Line 1:		Line 1:
	'''This article is still being developed - released earlier to align with Department seminar.'''		'''This article is still being developed - released earlier to align with Department seminar.'''

	In python, parallelization is a bit more of a “wild west” in terms of the number of packages that can help you parallelize your processes (~~for a long .~~ This wiki will attempt to present minor examples of some along with an outline of things to think about in selecting the library you end up using. In general, the process in python involves		In python, parallelization is a bit more of a “wild west” in terms of the number of packages that can help you parallelize your processes (This wiki will attempt to present minor examples of some along with an outline of things to think about in selecting the library you end up using. In general, the process in python involves

	# defining a function that captures what you want to have parallelized		# defining a function that captures what you want to have parallelized