Stat DS Computing Wiki - User contributions [en]

What resources when

2025-07-29T17:02:59Z

Ecucuzze:

To try to answer the common question “what resources do I have access to and when should I use them”, below we present to approaches to answering the question. The first of which focuses on what resources exist and what they should be used for and the second tries to answer resource needs / current computing problems with resources to fit those needs/ solve those problems.

After you’re found the resource you think you’d like to use (or even when you’re still deciding), we strongly encourage you to attend an computing office hour to help discuss all the things you should be thinking about and get you on the right track quickly.

= What computing resources do you have access to =

== Your personal computer ==

Your personal computer is great for small jobs that take less than 30 minutes, uses few cores and not too much RAM.

'''Pro:'''

#Naturally the easiest tool to use since it’s yours. Best for short jobs that require only a few cores.

'''Cons:'''

#
capacity:

#*few cores: 4-8 cores common on laptops<ref>From [http://laptopmag.com/articles/Cpu-comparison laptopmag]</ref> (multiply processes per cors means = 8-16 “cores”).
#*low RAM: 4-32 GB of common on laptops<ref>From [http://digitaltrends.com/computing/how-much-ram digitaltrends]</ref>, used across all cores
#*“no” GPUs: you don’t have GPUs that are optimized for Deep Learning

#
other usage: You often want to use your computer for other things (and it’s built for that exact thing). It’s possible you or someone you know has has had to leave their laptop at home for the day/ run lode oernight When they need to run something intensive -OR- long. But you need that laptop for other work / homework…

The below XKCD comic doesn’t have to be you… (though breaks are good):

[[File:Compiling.png|thumb|none|https://xkcd.com/303/]]

'''Conclusion:''' Your personal computer isn’t the best for when doing a lot of computing and its easy to put jobs elsewhere!

'''Common use-cases'''

#Most class projects
#Non-data problems
#Non-complex structure problems

'''Associated Articles''':

*[[Parallel_computing|parallelization]]

== The <code>hydras</code> (the department servers) ==

The departmental servers should be seen as a “sandbox”. You can experiment with lots of things on the servers and it doesn’t cost you much to do. The hydras have a decent amount of RAM (much more than your personal computer), but often it won’t run a lot faster than if you put you’re computer aside and did the same thing on it (though a large pro is you don’t have to stop using your own computer when running jobs on the hydras).

'''Pros:'''

#Best for long but smaller core jobs
#Decent amount of RAM per machine (64 GB, or 128 GB on hydra7 and hydra8)
#Very low barrier to entry (aka easy to get started)
#File management system shares across different hydra servers

'''Cons:'''

#usage: The department server is a bit of a “wild-west” in terms of usage (it varies how many people use it at a given time)
#cores per server: the number of cores per server is only 12

'''Conclusion:''' Great place to start for more intensive computing.

'''Associated Articles''':

*[[Linux_and_department_servers|Linux and department servers]]
'''Server Monitoring'''

View current Hydra resource usage - [https://info.stat.cmu.edu/grafana/d/fdkb6l4klj400d/hydra-cluster?orgId=1 Hydra Cluster Usage Monitor]

== PSC (Pittsburgh supercomputer center) ==

The Pittsburgh supercomputer center provides supercomputers than have massive amounts of cores (but also can find some with large RAMs and GPUs). Generally speaking, the supercomputing resources are often used when you have extremely parallelizable code that would take a very long time if you ran it on just a few cores (e.g. your computer or a few hydras).

'''Pros:'''

#A massive amount of cores and RAM that can be leveraged for extremely parallelizable computing
#Easier to access in the statistics department

'''Cons:'''

#Requires learning of new job management system (though quick to learn - see [Slurm])
#Some options to access the resources can take time or require going through your advisor
#''currently (2021)'' has been having problems with servers crashing

'''Associated Articles''':

*[[PSC|PSC]]
*[[Slurm|Slurm]]

== GPU Computing ==
The Statistics & Data Science Department offers two main GPU computing environments.

'''Ghidorahs'''

* 3 physical servers
* RTX 4090 GPUs
* Connected to the Stats network share
* Your statistics directory account will grant you access

'''Wright'''

* For a detail overview, please check [[GPU Cluster|here]]
* You must request access
* The Statistics network share is not accessible

= Computing resource flowchart =

These flow charts try to help capture what’s available and when you might use them.

Visit the links below to view current computing resource usage. Select a server with the least amount of users and lowest resource usage.

* [https://info.stat.cmu.edu/grafana/d/fdkb6l4klj400d/hydra-cluster?orgId=1 Hydra Cluster Usage Monitor]

* [https://info.stat.cmu.edu/grafana/d/edkar1rgy3l6od/ghidorah-cluster?orgId=1 Ghidorah Cluster Usage Monitor]
[[File:ProblemsOnYourPersonalComputerUpdated.png|center|thumb|1000x1000px]]

From [http://laptopmag.com/articles/Cpu-comparison laptopmag]
      2. ↑
From [http://digitaltrends.com/computing/how-much-ram digitaltrends]

File:ProblemsOnYourPersonalComputerUpdated.png

2025-07-29T17:01:45Z

Ecucuzze:

Problems on Your Personal Computer

What resources when

2025-07-29T16:25:53Z

Ecucuzze: /* Azure */ removed Azure

To try to answer the common question “what resources do I have access to and when should I use them”, below we present to approaches to answering the question. The first of which focuses on what resources exist and what they should be used for and the second tries to answer resource needs / current computing problems with resources to fit those needs/ solve those problems.

After you’re found the resource you think you’d like to use (or even when you’re still deciding), we strongly encourage you to attend an computing office hour to help discuss all the things you should be thinking about and get you on the right track quickly.

= What computing resources do you have access to =

== Your personal computer ==

Your personal computer is great for small jobs that take less than 30 minutes, uses few cores and not too much RAM.

'''Pro:'''

#Naturally the easiest tool to use since it’s yours. Best for short jobs that require only a few cores.

'''Cons:'''

#
capacity:

#*few cores: 4-8 cores common on laptops<ref>From [http://laptopmag.com/articles/Cpu-comparison laptopmag]</ref> (multiply processes per cors means = 8-16 “cores”).
#*low RAM: 4-32 GB of common on laptops<ref>From [http://digitaltrends.com/computing/how-much-ram digitaltrends]</ref>, used across all cores
#*“no” GPUs: you don’t have GPUs that are optimized for Deep Learning

#
other usage: You often want to use your computer for other things (and it’s built for that exact thing). It’s possible you or someone you know has has had to leave their laptop at home for the day/ run lode oernight When they need to run something intensive -OR- long. But you need that laptop for other work / homework…

The below XKCD comic doesn’t have to be you… (though breaks are good):

[[File:Compiling.png|thumb|none|https://xkcd.com/303/]]

'''Conclusion:''' Your personal computer isn’t the best for when doing a lot of computing and its easy to put jobs elsewhere!

'''Common use-cases'''

#Most class projects
#Non-data problems
#Non-complex structure problems

'''Associated Articles''':

*[[Parallel_computing|parallelization]]

== The <code>hydras</code> (the department servers) ==

The departmental servers should be seen as a “sandbox”. You can experiment with lots of things on the servers and it doesn’t cost you much to do. The hydras have a decent amount of RAM (much more than your personal computer), but often it won’t run a lot faster than if you put you’re computer aside and did the same thing on it (though a large pro is you don’t have to stop using your own computer when running jobs on the hydras).

'''Pros:'''

#Best for long but smaller core jobs
#Decent amount of RAM per machine (64 GB, or 128 GB on hydra7 and hydra8)
#Very low barrier to entry (aka easy to get started)
#File management system shares across different hydra servers

'''Cons:'''

#usage: The department server is a bit of a “wild-west” in terms of usage (it varies how many people use it at a given time)
#cores per server: the number of cores per server is only 12

'''Conclusion:''' Great place to start for more intensive computing.

'''Associated Articles''':

*[[Linux_and_department_servers|Linux and department servers]]
'''Server Monitoring'''

View current Hydra resource usage - [https://info.stat.cmu.edu/grafana/d/fdkb6l4klj400d/hydra-cluster?orgId=1 Hydra Cluster Usage Monitor]

== PSC (Pittsburgh supercomputer center) ==

The Pittsburgh supercomputer center provides supercomputers than have massive amounts of cores (but also can find some with large RAMs and GPUs). Generally speaking, the supercomputing resources are often used when you have extremely parallelizable code that would take a very long time if you ran it on just a few cores (e.g. your computer or a few hydras).

'''Pros:'''

#A massive amount of cores and RAM that can be leveraged for extremely parallelizable computing
#Easier to access in the statistics department

'''Cons:'''

#Requires learning of new job management system (though quick to learn - see [Slurm])
#Some options to access the resources can take time or require going through your advisor
#''currently (2021)'' has been having problems with servers crashing

'''Associated Articles''':

*[[PSC|PSC]]
*[[Slurm|Slurm]]

== GPU Computing ==
The Statistics & Data Science Department offers two main GPU computing environments.

'''Ghidorahs'''

* 3 physical servers
* RTX 4090 GPUs
* Connected to the Stats network share
* Your statistics directory account will grant you access

'''Wright'''

* For a detail overview, please check [[GPU Cluster|here]]
* You must request access
* The Statistics network share is not accessible

= Computing resource flowchart =

These flow charts try to help capture what’s available and when you might use them.

Visit the links below to view current computing resource usage. Select a server with the least amount of users and lowest resource usage.

* [https://info.stat.cmu.edu/grafana/d/fdkb6l4klj400d/hydra-cluster?orgId=1 Hydra Cluster Usage Monitor]

* [https://info.stat.cmu.edu/grafana/d/edkar1rgy3l6od/ghidorah-cluster?orgId=1 Ghidorah Cluster Usage Monitor]
== [[File:ProblemsOnYourPersonalComputer11.png|706x706px]]==

From [http://laptopmag.com/articles/Cpu-comparison laptopmag]
      2. ↑
From [http://digitaltrends.com/computing/how-much-ram digitaltrends]

Azure getting started

2025-07-29T16:24:58Z

Ecucuzze: Removed any reference to the department's azure resources. Made it generic

= Azure =

== Respectiful use of Azure resources ==

Ensure you’re not wasting precious department Azure credits - when you are not using your virtual machine, please shut it down (and then relaunch it when you need to use it). You can shut-down and relauch a server on the resource page. The overview for your virtual machine might look something like the image below. To turn off your machine click the “Stop” button on the “Overview” tab, and to launch your machine click the “Start” button. Every time you launch your machine it’s likely that the IP address will have changed, so use the “Connect” button to figure out the new IP address (though it is also listed on the Overview page). The “Connect” page (which appears in the second image) will also remind you of what your username for the virtual machine is.

[[File:1600px-Azure on off.png|frameless|800px|turning on and off server to save department money]]

[[File:1600px-Azure connect.png|frameless|800px|connecting to your server's IP address when it changes]]

= Getting started: create an Azure virtual machine =

== Creating a Resource ==

You will need to have an established Azure account, which no longer can be done through department resources as of Summer 2025. Creating a virtual on Azure can be a bit complicated (which is why other resources like the department servers are available for certain types of computing).

The first thing to do is click the “create a resource” button on your azure home page (should look something like this:)

[[File:Azure create new.png|frameless|300px|create new server button]]

You can also do so in the <code>All resources</code> > <code>+ Create</code>

 

----

For statistical computing we select a resource from the <code>AI + Machine Learning</code> tab on the left side and then select <code>Data Science Virtual Machine - Ubuntu 18.04</code> for a linux virtual machine. This selection will look something like the image below:

[[File:Azure select resource type.png|frameless|700px|Selecting which type of virtual server to use]]

=== Defining your Resource ===

There are a lot of options to create your resource, but you can ignore a lot of these.

==== Basics ====

Below is 2 images of the first “Basics” page. You will need to

#Fill out / select your “Resource group”
#Define your “Virtual machine name” (might we recommend (username)-(CPU/GPU)-cores(#)-(RAM)-(index). For example “bpleroy-GPU-cores4-27-1”.
#Select the size of the machine (relative to it’s cores and types of cores). You will see available options. You can also click "see all images" to explore a list of options. If you are looking for GPU in this list filter by "family" and "GPU" for the family.
#Finally, we recommend using a ssh public key (which will be downloaded at the end of the set-up process). We recommend you change your “Username” to your andrew ID but you can leave it as “azureuser” if you’d like (we recommend such things since you may forget and it's natural to have it consistent across your servers).

[[File:1400px-Azure basics 1.png|frameless|700px|setting up new server, basic page (1)]]

[[File:1400px-Azure basics 2.png|frameless|700px|setting up new server, basic page (2)]]

==== Disks ====

The second pages is about “Disks” although creating storage on Azure is complicated it’s a very necessary procedure. Click “Create and attache a new disk”. You will then have to select the right size. Generally, disks smaller than 512 GB are cheap for the department but 1 TB + are more costly. You personal computer likely has between 128 - 512 GB (maybe 1TB), but it’s really unlikely you’ll need too much for your project.

[[File:1400px-Azure disk creation.png|frameless|700px|creating a disk]]

 

==== Network and Management ====

There isn’t really anything necessary for you to do on these tabs.

==== Advanced ====

If you are using GPUs you’ll need to add the NVDIA extension for GPUs.

==== Tags are also not necessary ====

==== Done? ====

You did it! Save your ssh key (<code>.pem</code> file) in a location where you plan to call the server from. Also note that you will '''still''' need to correctly link your data disk you just created (which is discussed in the next step).

=== Connecting your data disk ===

This process is slightly complex (and can have implications if you don’t do it correctly). We recommend you follow the [https://docs.microsoft.com/en-us/azure/virtual-machines/linux/attach-disk-portal azure docs].

==== step 1: finding and identifying your disk ====

on the server run something like:

 
<syntaxhighlight lang="bash">lsblk -o NAME,HCTL,SIZE,MOUNTPOINT | grep -i "sd"</syntaxhighlight>

you should see a list something like this:

 
<syntaxhighlight lang="bash">sda 3:0:0:0 128G
sdb 0:0:0:0 150G
├─sdb1 149.9G /
├─sdb14 4M
└─sdb15 106M /boot/efi
sdc 1:0:1:0 176G
└─sdc1 176G /mnt</syntaxhighlight>

now, on portal for the same VM, look at your disks (left side: Settings > Disks) and examine the size of your disk you added (for mine it’s 128G). As such we can conclude that my disk that needs mounting is <code>sda</code> (this might not be the same for you - and is not the same in the azure docs).

==== step 2: partitioning a new disk ====

Copied form the azure docs (but changed for my disk name)

 
<syntaxhighlight lang="bash">sudo parted /dev/sda --script mklabel gpt mkpart xfspart xfs 0% 100%
sudo mkfs.xfs /dev/sda1
sudo partprobe /dev/sda1</syntaxhighlight>

==== step 3: mount the disk ====

Create a directory to mount the file system using mkdir. The following example creates a directory at <code>/datadrive</code>:

 
<syntaxhighlight lang="bash">sudo mkdir /datadrive</syntaxhighlight>

Use mount to then mount the filesystem. The following example mounts the <code>/dev/sda1</code> partition to the <code>/datadrive</code> mount point:

 
<syntaxhighlight lang="bash">sudo mount /dev/sda1 /datadrive</syntaxhighlight>

''Now we have'':

 
<syntaxhighlight lang="bash">lsblk -o NAME,HCTL,SIZE,MOUNTPOINT | grep -i "sd"</syntaxhighlight>
<syntaxhighlight lang="bash">sda 3:0:0:0 128G
└─sda1 128G /datadrive
sdb 0:0:0:0 150G
├─sdb1 149.9G /
├─sdb14 4M
└─sdb15 106M /boot/efi
sdc 1:0:1:0 176G
└─sdc1 176G /mnt</syntaxhighlight>

==== step 4: … ====

 
<syntaxhighlight lang="bash">sudo blkid</syntaxhighlight>
<syntaxhighlight lang="bash">/dev/sdb1: LABEL="cloudimg-rootfs" UUID="1fe704c3-9cec-4cb5-b634-2e46f6bd9f90" TYPE="ext4" PARTUUID="318ec118-9248-4203-9cd5-7cb8c36f04d6"
/dev/sdb15: LABEL="UEFI" UUID="52A4-029B" TYPE="vfat" PARTUUID="2332d2b4-222e-425d-866e-227b42f3979d"
/dev/sdc1: UUID="9cc09cc7-c65d-4fe4-a397-75679a438c1c" TYPE="ext4" PARTUUID="2b584de2-01"
/dev/sda1: UUID="a2ba271d-e5a4-4416-84d2-b9408864b53b" TYPE="xfs" PARTLABEL="xfspart" PARTUUID="77452866-0b43-4485-9956-673377498608"</syntaxhighlight>

==== step 5: creating alias link to datadrive ====

To create a symbolic link to your a <code>workspace</code> folder to the <code>/datadrive</code> you can do the following.

 
<syntaxhighlight lang="bash">ln -s /datadrive ~/workspace </syntaxhighlight>

Afterwards, everything in the <code>workspace</code> will actually be stored on your new <code>/datadrive</code>.

==== step 6: making sure you have correct permissions ====

<code>ls -al /</code> (might show you don’t have access (write, read, execute) to the /datadrive), so we did

<code>sudo chmod 777 datadrive</code> …

[https://www.hostinger.com/tutorials/how-to-create-symbolic-links-in-linux/ https://www.hostinger.com/tutorials/how-to-create-symbolic-links-in-linux/]

=== Connecting to your server and moving files ===

To connect to your server you will need to know (1) where you're ssh key is (often called <code>_server_name_.pem</code> (this should have been downloaded at the end of the creation of your server). '''Note that the first time your connect you need to make sure you have the correct permissions - see the connect page for the correct <code>cmod</code> command you need to use!''' (2) your username defined in the [[Azure_getting_started#Basics|Basics]] step

You will also need to know what IP address your server is currently using. This can before on the connect page of your server (as seen in the second image in this article). Then you're type something like

 
<syntaxhighlight lang="bash">
ssh -i _my_ssh_key_.pem _user_name_@_IP_address_
</syntaxhighlight>

To pass files to and from your server with <code>scp</code>, we will imagine you are on your home computer. Then do as follows:

#'''server to local''' <syntaxhighlight lang="bash">
scp -i _my_ssh_key_.pem _user_name_@_IP_address_:/full/path/to/files/on/server/my_script.R /local/path/to/files/on/local/computer/
</syntaxhighlight>

#'''local to server''' <syntaxhighlight lang="bash">
scp -i _my_ssh_key_.pem /local/path/to/files/on/local/computer/my_script.R _user_name_@_IP_address_:/full/path/to/files/
</syntaxhighlight>

To pass directories (and do the full thing you'll need to do the same as above but add

 
<syntaxhighlight lang="bash">
scp -i _my_ssh_key.pem -r ...
</syntaxhighlight>

Main Page

2025-07-29T16:22:02Z

Ecucuzze: Created an archive section and moved azure there

Anyone can read this wiki.  If you want to write/edit stuff in the wiki, you will need to log in with your account provided by the Statistics & Data Science Department computing group. This Wiki is intended to be a repository for instruction, information and ideas related to the many computing environments available to members and affiliates of the Statistics & Data Science Department.

== Getting to know the computing resources & getting support ==

If you're interested in knowing about what resources you have access to (and/or) what resources you have access to that might solve a problem you're having, please visit our [[What_resources_when|"what resources when" article]].

To get support that is beyond this website please attend office hours below and/or email [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu] for support (for more details about how the support ticketing works through [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu] + ServiceNow please see the [[Service_now|ServiceNow]] article).

Reset your StatDS password here - https://util.stat.cmu.edu/passwordreset/

== Department Contacts ==

• [mailto:events@stat.cmu.edu events@stat.cmu.edu]: seminars, conferences & workshops, and calendar updates

• [mailto:news@stat.cmu.edu news@stat.cmu.edu]: dept/dept member-related public news & announcements

• [mailto:lists@stat.cmu.edu lists@stat.cmu.edu]: department email lists

• [mailto:webmaster@stat.cmu.edu webmaster@stat.cmu.edu]: website updates

• [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu]: computer/tech requests

• [mailto:rooms@stat.cmu.edu rooms@stat.cmu.edu]: request space for meetings and TA office hours

• [mailto:reimbursements@stat.cmu.edu reimbursements@stat.cmu.edu]: travel and reimbursement requests

== Computing Office Hours ==

These office hour opportunities provide a dedicated time for you to come ask questions about research computing.  These include code advice, how to access certain resources, what the best resources are for your project, or to learn about new connections to help you progress in your education and research. Staff Office hour times will be sent out each week on Monday as a calendar invite; accept any of the invitations to add it to your personal calendar. PhD TA Office Hours are below, unless a different time is communicated.

#PhD TA Office Hours for the first half of summer 2025 are held by Elizabeth Cucuzzella in FMS Room 320 on Mondays from 2:30 pm to 4:30 pm. PhD students can also message on the #computing channel on slack.
#Staff Office Hours are held by appointment by Carl Skipper in BH 129G. Schedule via [mailto:lists@stat.cmu.edu email].

See [[Computing_office_hour_usage|computing office hour usage]] for what office hours can be used for.

== Computing Wiki Pages ==

=== Managing servers and environments ===

==== Introductory Tutorials ====
[[Basic Linux Commands|Introduction to Basic Linux Commands]]

==== Virtual environments ====
*[[Virtual Environment Introduction|Why use virtual environments?]]
*R's [[Renv]]
*Python's [[Python_virtual_environments|virtual environments (2 options)]]
*Old instructions for [[Conda with Python (Reference Only)|conda with Python]]

==== Containers ====
*[[Introduction to containers]]
*[[Docker, and why-you-should-use-it-for-your-research]]
*[[Tutorials|Basic tutorial]]
*[[Running Docker locally and on Hydras]]

=== Department computing resources ===

==== Printing ====
*[[Department_Printers|Department Printers]]

==== Web Hosting ====
#[[How to host your webpage under stat.cmu.edu]]
#[[Bulding your website with Hugo]]

==== Servers ====
*[[Linux_and_department_servers|Department Servers]]. This wiki introduces you to the '''hydra''' servers, basic Linux commands, and more.
*#[[Terminal_multiplexers|Terminal multiplexers (screen and tmux)]]. This wiki presents tools for use on the ''hydra'' that can allow you to run jobs even when your not on the server, and much much more!
*#[[Python on the server]]. This wiki introduces uses command line approaches to running <code>python</code> on a virtual machine, and how to use jupyter notebooks through the server as well.
*#[[R on the server]]. This wiki introduces uses command line approaches to running <code>R</code> on a virtual machine

*[[Department GPU Servers]], for small jobs needing 1 GPU.
*[[GPU Cluster]], for larger jobs that may need multiple GPUs and many CPUs.
*[[PSC]], the Pittsburgh Supercomputing Center. Free accounts and usage credits are available
*#[[ACCESS]]
*#[[Slurm]]. This wiki introduces the job management system uses at PSC.
*#[[Parallelization_on_PSC/Bridges-2]]
*#[[Access PSC compute nodes directly from VSCode]]
*[[Open OnDemand]]

=== Development guides ===

====[[Parallel_computing|Parallelization]]====
*[[Parallelization_in_r|R-specific parallelization]]
*[[Parallelization_in_python|Python-specific parallelization]]

==== Package development ====
*[[Developing_R_packages|R package development]]
*[[Developing_python_packages|Python package development]]
[[Terminal automation]]. This wiki introduces approaches to automate Linux/OS command line calls.
*[[Bash files]]. This wiki presents potential uses of Bash files including automated file/folder creation, parallelization and much more!
*[[Make]]. This wiki presents potential uses of <code>make</code> and Makefiles including automated workflow and dependency tracking
*[[Jumping to frequently used folders and automating environment setups]]

==== Version Control ====
See the guide provided in the site of [https://36-750.github.io/tools/version-control/ 36750 - Statistical Computing]

==== Reference management ====
#Zotero
#Mendeley

==== Integrated Development Environments (IDEs) ====
#[[VSCode|VSCode]]
#PyCharm
#RStudio
#Eclipse

==== Text Editors ====
#Vim
#Nano
#Emacs
#Atom
#Sublime

See also the guide provided in the site of [https://36-750.github.io/tools/editors/ 36750 - Statistical Computing]

==== AI Tools ====
* [[ChatGPT and other LLMs]]

*[[github_copilot|GitHub Copilot]]

==== Miscellaneous====
#[[Windows_Subsystem_for_Linux_(WSL)_and_Python|Windows Subsystem for Linux (WSL) and Python]]
#[[Amphetamine|Amphetamine: How to keep your computer awake]]

===General Department Computing Resources===

*Fall 2021 [[Computing_seminar_resources|Computing Seminar Resources]] (video and slides)

==Ideas for the Wiki==

Please see the [[TODO|TODO page]] to post suggestions and errata.

==Using the Wiki==

*Here is some [[Generic_Wiki_Information|Generic Wiki Information]] that you may be interested in, such as a users' guide and other resources for Wiki users.

== Archived Pages ==
These area pages that are potentially useful for some users, but no longer reflect the computing resources of the department writ large.

Azure
#[[Azure_getting_started|Azure: getting started]]. This wiki walks you through the steps to create a virtual machine through Azure
#[[How to obtain a virtual machine with a GPU]]
#[[Attach a disk to a virtual machine]]

Open OnDemand

2025-07-29T15:20:26Z

Ecucuzze: /* Getting Started */

Open OnDemand provides easy, remote web access to supercomputers and computing clusters, without the need for a command-line interface. Instead of accessing the hydra and ghidorah clusters via your terminal, Open OnDemand provides a user interface for accessing and modifying files and submitting jobs. No installation is required -- everything can be accessed within the browser.

=== Getting Started ===
To access Open OnDemand, login [https://ood.stat.cmu.edu/dex/auth/ldap/login?back=&state=iqxvp3cbpitdiomazu5eih4nc here] with your StatDS username and password. This is the same username and password you would use to access the clusters via ssh on your terminal.

After logging in, you should be redirected to your file system. You can navigate through your directories just as you would on your desktop. A nice feature of Open OnDemand is you can automatically upload and download files from your desktop to the department's file system; the secure file transfer is done automatically in a user-friendly manner.

=== Editing Files ===
To create a new file or directory, simply click "New File" or "New Directory" in the upper right hand corner of the screen.

To edit existing files, navigate to the file in the file system. Click on the dropdown three vertical dots to the right of the file's name. An options list will appear that includes a "View" mode and an "Edit" mode. Clicking view will allow you to view the contents of the file without manipulating them. Edit will redirect you to another tab where you can directly edit the contents of the file. Click the "Save" button on the lefthand side of the screen to save edits; when this button is grayed out, it means there are no additional changes to save.

=== Accessing the Clusters ===
To access the head node via the terminal, navigate to the "Clusters" drop down in the top left of the screen, and select "Statistics and Data Science Shell Access." The shell allows you to access your files within a terminal on the head compute node. Do not run jobs within this terminal. Beneath the cluster selection, you can view the system status which tells you how many nodes, processors, and GPUs are currently available, as well as how many jobs are running and queued. This will help inform how long a job you've created might take to run on the system/when it will begin running.

=== Submitting and Tracking Jobs ===

==== Default Template ====
To submit or compose a job, navigate to the "Jobs" drop down in the top left of the screen, and select "Job Composer." This will take you to the job composer page, where you can select the "New Job" drop down. To start from the default template, just click "From Default Template" and in the right hand panel, scroll down to where the .sh script is shone and click "Open Editor" to edit the script itself. Once you are ready to submit the job, simply click the job in your list of potential jobs, and click submit. The job will show a status of "Queued," "Running," or "Completed" and you can view output files in the right hand panel once it has completed running.

==== Hydras and Ghidorah Templates ====
If you click "From Template," you can choose to submit a job on the hydras or ghidorahs. Simply pick the template you want to use -- either Hydra Job for the hydras or Ghidorah Job for the ghidorahs -- and click "Create New Job." You can then modify the submit.sh script as before to select ''which'' hydra or ghidorah you want to use as follows:<syntaxhighlight lang="bash">
#!/bin/sh
#SBATCH --partition Ghidorahs

# #Select a specific Hydra node to run job on.
# #Un-comment the line below and replace ghidorahXX with the appropriate ghidorah server ~ ghidorah1 in this case.
#SBATCH --nodelist=ghidorah1

# #Number of tasks.
#SBATCH --ntasks 1

# #Number of CPUs per tasks.
#SBATCH --cpus-per-task 1

# #Amount of RAM
#SBATCH --mem=4G

# #Output file for script.
#SBATCH --output=%j_output.txt

# #Error file for script.
#SBATCH --error=%j_error.txt

python3 examplepython.py
</syntaxhighlight>The <code>#SBATCH --nodelist=ghidorah1</code> selects the 1st ghidorah to run the python script on. Change this to <code>#SBATCH --nodelist=hydra1</code> for the hydras and change the first SBATCH line to Hydras (<code>#SBATCH --partition Hydras</code>).

==== Submitting a Script from a Specified Path ====
You can also specify a path to a job you have already created. Click "From Specified Path" and enter the source path, i.e. the path to the directory containing your .sh file. Once you have done that, you can specify the name of the script under Script name, i.e. the name of the .sh file in the directory you've navigated to. You can then click "Run" just as you have with your other jobs.

Open OnDemand

2025-07-16T15:04:11Z

Ecucuzze:

Open OnDemand provides easy, remote web access to supercomputers and computing clusters, without the need for a command-line interface. Instead of accessing the hydra and ghidorah clusters via your terminal, Open OnDemand provides a user interface for accessing and modifying files and submitting jobs. No installation is required -- everything can be accessed within the browser.

=== Getting Started ===
To access Open OnDemand, login [https://ood.stat.cmu.edu/dex/auth/ldap/login?back=&state=oiswjnltgm3e46hgcld2kyple here] with your StatDS username and password. This is the same username and password you would use to access the clusters via ssh on your terminal.

After logging in, you should be redirected to your file system. You can navigate through your directories just as you would on your desktop. A nice feature of Open OnDemand is you can automatically upload and download files from your desktop to the department's file system; the secure file transfer is done automatically in a user-friendly manner.

=== Editing Files ===
To create a new file or directory, simply click "New File" or "New Directory" in the upper right hand corner of the screen.

To edit existing files, navigate to the file in the file system. Click on the dropdown three vertical dots to the right of the file's name. An options list will appear that includes a "View" mode and an "Edit" mode. Clicking view will allow you to view the contents of the file without manipulating them. Edit will redirect you to another tab where you can directly edit the contents of the file. Click the "Save" button on the lefthand side of the screen to save edits; when this button is grayed out, it means there are no additional changes to save.

=== Accessing the Clusters ===
To access the clusters via the terminal, navigate to the "Clusters" drop down in the top left of the screen, and select "Statistics and Data Science Shell Access." The shell access is built for both the ghidorahs and the hydras. Beneath the cluster selection, you can view the system status which tells you how many nodes, processors, and GPUs are currently available, as well as how many jobs are running and queued. This will help inform how long a job you've created might take to run on the system/when it will begin running.

=== Submitting and Tracking Jobs ===

==== Default Template ====
To submit or compose a job, navigate to the "Jobs" drop down in the top left of the screen, and select "Job Composer." This will take you to the job composer page, where you can select the "New Job" drop down. To start from the default template, just click "From Default Template" and in the right hand panel, scroll down to where the .sh script is shone and click "Open Editor" to edit the script itself. Once you are ready to submit the job, simply click the job in your list of potential jobs, and click submit. The job will show a status of "Queued," "Running," or "Completed" and you can view output files in the right hand panel once it has completed running.

==== Hydras and Ghidorah Templates ====
If you click "From Template," you can choose to submit a job on the hydras or ghidorahs. Simply pick the template you want to use -- either Hydra Job for the hydras or Ghidorah Job for the ghidorahs -- and click "Create New Job." You can then modify the submit.sh script as before to select ''which'' hydra or ghidorah you want to use as follows:<syntaxhighlight lang="bash">
#!/bin/sh
#SBATCH --partition Ghidorahs

# #Select a specific Hydra node to run job on.
# #Un-comment the line below and replace ghidorahXX with the appropriate ghidorah server ~ ghidorah1 in this case.
#SBATCH --nodelist=ghidorah1

# #Number of tasks.
#SBATCH --ntasks 1

# #Number of CPUs per tasks.
#SBATCH --cpus-per-task 1

# #Amount of RAM
#SBATCH --mem=4G

# #Output file for script.
#SBATCH --output=%j_output.txt

# #Error file for script.
#SBATCH --error=%j_error.txt

python3 examplepython.py
</syntaxhighlight>The <code>#SBATCH --nodelist=ghidorah1</code> selects the 1st ghidorah to run the python script on. Change this to <code>#SBATCH --nodelist=hydra1</code> for the hydras and change the first SBATCH line to Hydras (<code>#SBATCH --partition Hydras</code>).

==== Submitting a Script from a Specified Path ====
You can also specify a path to a job you have already created. Click "From Specified Path" and enter the source path, i.e. the path to the directory containing your .sh file. Once you have done that, you can specify the name of the script under Script name, i.e. the name of the .sh file in the directory you've navigated to. You can then click "Run" just as you have with your other jobs.

Open OnDemand

2025-07-16T14:53:27Z

Ecucuzze: Created page with "Open OnDemand provides easy, remote web access to supercomputers and computing clusters, without the need for a command-line interface. Instead of accessing the hydra and ghid..."

Main Page

2025-07-16T13:47:53Z

Ecucuzze:

Anyone can read this wiki.  If you want to write/edit stuff in the wiki, you will need to log in with your account provided by the Statistics & Data Science Department computing group. This Wiki is intended to be a repository for instruction, information and ideas related to the many computing environments available to members and affiliates of the Statistics & Data Science Department.

== Getting to know the computing resources & getting support ==

If you're interested in knowing about what resources you have access to (and/or) what resources you have access to that might solve a problem you're having, please visit our [[What_resources_when|"what resources when" article]].

To get support that is beyond this website please attend office hours below and/or email [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu] for support (for more details about how the support ticketing works through [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu] + ServiceNow please see the [[Service_now|ServiceNow]] article).

Reset your StatDS password here - https://util.stat.cmu.edu/passwordreset/

== Department Contacts ==

• [mailto:events@stat.cmu.edu events@stat.cmu.edu]: seminars, conferences & workshops, and calendar updates

• [mailto:news@stat.cmu.edu news@stat.cmu.edu]: dept/dept member-related public news & announcements

• [mailto:lists@stat.cmu.edu lists@stat.cmu.edu]: department email lists

• [mailto:webmaster@stat.cmu.edu webmaster@stat.cmu.edu]: website updates

• [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu]: computer/tech requests

• [mailto:rooms@stat.cmu.edu rooms@stat.cmu.edu]: request space for meetings and TA office hours

• [mailto:reimbursements@stat.cmu.edu reimbursements@stat.cmu.edu]: travel and reimbursement requests

== Computing Office Hours ==

These office hour opportunities provide a dedicated time for you to come ask questions about research computing.  These include code advice, how to access certain resources, what the best resources are for your project, or to learn about new connections to help you progress in your education and research. Staff Office hour times will be sent out each week on Monday as a calendar invite; accept any of the invitations to add it to your personal calendar. PhD TA Office Hours are below, unless a different time is communicated.

#PhD TA Office Hours for the first half of summer 2025 are held by Elizabeth Cucuzzella in FMS Room 320 on Mondays from 2:30 pm to 4:30 pm. PhD students can also message on the #computing channel on slack.
#Staff Office Hours are held by appointment by Carl Skipper in BH 129G. Schedule via [mailto:lists@stat.cmu.edu email].

See [[Computing_office_hour_usage|computing office hour usage]] for what office hours can be used for.

== Computing Wiki Pages ==

=== Managing servers and environments ===

==== Introductory Tutorials ====
[[Basic Linux Commands|Introduction to Basic Linux Commands]]

==== Virtual environments ====
*[[Virtual Environment Introduction|Why use virtual environments?]]
*R's [[Renv]]
*Python's [[Python_virtual_environments|virtual environments (2 options)]]
*Old instructions for [[Conda with Python (Reference Only)|conda with Python]]

==== Containers ====
*[[Introduction to containers]]
*[[Docker, and why-you-should-use-it-for-your-research]]
*[[Tutorials|Basic tutorial]]
*[[Running Docker locally and on Hydras]]

=== Department computing resources ===

==== Printing ====
*[[Department_Printers|Department Printers]]

==== Web Hosting ====
#[[How to host your webpage under stat.cmu.edu]]
#[[Bulding your website with Hugo]]

==== Servers ====
*[[Linux_and_department_servers|Department Servers]]. This wiki introduces you to the '''hydra''' servers, basic Linux commands, and more.
*#[[Terminal_multiplexers|Terminal multiplexers (screen and tmux)]]. This wiki presents tools for use on the ''hydra'' that can allow you to run jobs even when your not on the server, and much much more!
*#[[Python on the server]]. This wiki introduces uses command line approaches to running <code>python</code> on a virtual machine, and how to use jupyter notebooks through the server as well.
*#[[R on the server]]. This wiki introduces uses command line approaches to running <code>R</code> on a virtual machine

*[[Department GPU Servers]], for small jobs needing 1 GPU.
*[[GPU Cluster]], for larger jobs that may need multiple GPUs and many CPUs.
*[[PSC]], the Pittsburgh Supercomputing Center. Free accounts and usage credits are available
*#[[ACCESS]]
*#[[Slurm]]. This wiki introduces the job management system uses at PSC.
*#[[Parallelization_on_PSC/Bridges-2]]
*#[[Access PSC compute nodes directly from VSCode]]
*Azure
*#[[Azure_getting_started|Azure: getting started]]. This wiki walks you through the steps to create a virtual machine through Azure
*#[[How to obtain a virtual machine with a GPU]]
*#[[Attach a disk to a virtual machine]]
*[[Open OnDemand]]

=== Development guides ===

====[[Parallel_computing|Parallelization]]====
*[[Parallelization_in_r|R-specific parallelization]]
*[[Parallelization_in_python|Python-specific parallelization]]

==== Package development ====
*[[Developing_R_packages|R package development]]
*[[Developing_python_packages|Python package development]]
[[Terminal automation]]. This wiki introduces approaches to automate Linux/OS command line calls.
*[[Bash files]]. This wiki presents potential uses of Bash files including automated file/folder creation, parallelization and much more!
*[[Make]]. This wiki presents potential uses of <code>make</code> and Makefiles including automated workflow and dependency tracking
*[[Jumping to frequently used folders and automating environment setups]]

==== Version Control ====
See the guide provided in the site of [https://36-750.github.io/tools/version-control/ 36750 - Statistical Computing]

==== Reference management ====
#Zotero
#Mendeley

==== Integrated Development Environments (IDEs) ====
#[[VSCode|VSCode]]
#PyCharm
#RStudio
#Eclipse

==== Text Editors ====
#Vim
#Nano
#Emacs
#Atom
#Sublime

See also the guide provided in the site of [https://36-750.github.io/tools/editors/ 36750 - Statistical Computing]

==== AI Tools ====
* [[ChatGPT and other LLMs]]

*[[github_copilot|GitHub Copilot]]

==== Miscellaneous====
#[[Windows_Subsystem_for_Linux_(WSL)_and_Python|Windows Subsystem for Linux (WSL) and Python]]
#[[Amphetamine|Amphetamine: How to keep your computer awake]]

===General Department Computing Resources===

*Fall 2021 [[Computing_seminar_resources|Computing Seminar Resources]] (video and slides)

==Ideas for the Wiki==

Please see the [[TODO|TODO page]] to post suggestions and errata.

==Using the Wiki==

*Here is some [[Generic_Wiki_Information|Generic Wiki Information]] that you may be interested in, such as a users' guide and other resources for Wiki users.

Main Page

2025-07-11T14:31:24Z

Ecucuzze:

Anyone can read this wiki.  If you want to write/edit stuff in the wiki, you will need to log in with your account provided by the Statistics & Data Science Department computing group. This Wiki is intended to be a repository for instruction, information and ideas related to the many computing environments available to members and affiliates of the Statistics & Data Science Department.

== Getting to know the computing resources & getting support ==

If you're interested in knowing about what resources you have access to (and/or) what resources you have access to that might solve a problem you're having, please visit our [[What_resources_when|"what resources when" article]].

To get support that is beyond this website please attend office hours below and/or email [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu] for support (for more details about how the support ticketing works through [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu] + ServiceNow please see the [[Service_now|ServiceNow]] article).

Reset your StatDS password here - https://util.stat.cmu.edu/passwordreset/

== Department Contacts ==

• [mailto:events@stat.cmu.edu events@stat.cmu.edu]: seminars, conferences & workshops, and calendar updates

• [mailto:news@stat.cmu.edu news@stat.cmu.edu]: dept/dept member-related public news & announcements

• [mailto:lists@stat.cmu.edu lists@stat.cmu.edu]: department email lists

• [mailto:webmaster@stat.cmu.edu webmaster@stat.cmu.edu]: website updates

• [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu]: computer/tech requests

• [mailto:rooms@stat.cmu.edu rooms@stat.cmu.edu]: request space for meetings and TA office hours

• [mailto:reimbursements@stat.cmu.edu reimbursements@stat.cmu.edu]: travel and reimbursement requests

== Computing Office Hours ==

These office hour opportunities provide a dedicated time for you to come ask questions about research computing.  These include code advice, how to access certain resources, what the best resources are for your project, or to learn about new connections to help you progress in your education and research. Staff Office hour times will be sent out each week on Monday as a calendar invite; accept any of the invitations to add it to your personal calendar. PhD TA Office Hours are below, unless a different time is communicated.

#PhD TA Office Hours for the first half of summer 2025 are held by Elizabeth Cucuzzella in FMS Room 320 on Mondays from 2:30 pm to 4:30 pm. PhD students can also message on the #computing channel on slack.
#Staff Office Hours are held by appointment by Carl Skipper in BH 129G. Schedule via [mailto:lists@stat.cmu.edu email].

See [[Computing_office_hour_usage|computing office hour usage]] for what office hours can be used for.

== Computing Wiki Pages ==

=== Managing servers and environments ===

==== Introductory Tutorials ====
[[Basic Linux Commands|Introduction to Basic Linux Commands]]

==== Virtual environments ====
*[[Virtual Environment Introduction|Why use virtual environments?]]
*R's [[Renv]]
*Python's [[Python_virtual_environments|virtual environments (2 options)]]
*Old instructions for [[Conda with Python (Reference Only)|conda with Python]]

==== Containers ====
*[[Introduction to containers]]
*[[Docker, and why-you-should-use-it-for-your-research]]
*[[Tutorials|Basic tutorial]]
*[[Running Docker locally and on Hydras]]

=== Department computing resources ===

==== Printing ====
*[[Department_Printers|Department Printers]]

==== Web Hosting ====
#[[How to host your webpage under stat.cmu.edu]]
#[[Bulding your website with Hugo]]

==== Servers ====
*[[Linux_and_department_servers|Department Servers]]. This wiki introduces you to the '''hydra''' servers, basic Linux commands, and more.
*#[[Terminal_multiplexers|Terminal multiplexers (screen and tmux)]]. This wiki presents tools for use on the ''hydra'' that can allow you to run jobs even when your not on the server, and much much more!
*#[[Python on the server]]. This wiki introduces uses command line approaches to running <code>python</code> on a virtual machine, and how to use jupyter notebooks through the server as well.
*#[[R on the server]]. This wiki introduces uses command line approaches to running <code>R</code> on a virtual machine

*[[Department GPU Servers]], for small jobs needing 1 GPU.
*[[GPU Cluster]], for larger jobs that may need multiple GPUs and many CPUs.
*[[PSC]], the Pittsburgh Supercomputing Center. Free accounts and usage credits are available
*#[[ACCESS]]
*#[[Slurm]]. This wiki introduces the job management system uses at PSC.
*#[[Parallelization_on_PSC/Bridges-2]]
*#[[Access PSC compute nodes directly from VSCode]]
*Azure
*#[[Azure_getting_started|Azure: getting started]]. This wiki walks you through the steps to create a virtual machine through Azure
*#[[How to obtain a virtual machine with a GPU]]
*#[[Attach a disk to a virtual machine]]

=== Development guides ===

====[[Parallel_computing|Parallelization]]====
*[[Parallelization_in_r|R-specific parallelization]]
*[[Parallelization_in_python|Python-specific parallelization]]

==== Package development ====
*[[Developing_R_packages|R package development]]
*[[Developing_python_packages|Python package development]]
[[Terminal automation]]. This wiki introduces approaches to automate Linux/OS command line calls.
*[[Bash files]]. This wiki presents potential uses of Bash files including automated file/folder creation, parallelization and much more!
*[[Make]]. This wiki presents potential uses of <code>make</code> and Makefiles including automated workflow and dependency tracking
*[[Jumping to frequently used folders and automating environment setups]]

==== Version Control ====
See the guide provided in the site of [https://36-750.github.io/tools/version-control/ 36750 - Statistical Computing]

==== Reference management ====
#Zotero
#Mendeley

==== Integrated Development Environments (IDEs) ====
#[[VSCode|VSCode]]
#PyCharm
#RStudio
#Eclipse

==== Text Editors ====
#Vim
#Nano
#Emacs
#Atom
#Sublime

See also the guide provided in the site of [https://36-750.github.io/tools/editors/ 36750 - Statistical Computing]

==== AI Tools ====
* [[ChatGPT and other LLMs]]

*[[github_copilot|GitHub Copilot]]

==== Miscellaneous====
#[[Windows_Subsystem_for_Linux_(WSL)_and_Python|Windows Subsystem for Linux (WSL) and Python]]
#[[Amphetamine|Amphetamine: How to keep your computer awake]]

===General Department Computing Resources===

*Fall 2021 [[Computing_seminar_resources|Computing Seminar Resources]] (video and slides)

==Ideas for the Wiki==

Please see the [[TODO|TODO page]] to post suggestions and errata.

==Using the Wiki==

*Here is some [[Generic_Wiki_Information|Generic Wiki Information]] that you may be interested in, such as a users' guide and other resources for Wiki users.

Command Line Basics

2025-07-11T14:30:11Z

Ecucuzze:

This page serves as an introduction to basic commands for managing your file system on Linux. Much of what is reviewed on this page is introduced as part of the computing course in Immigration to Statistics.

These commands can be used on any operating system using Linux. To get started, navigate to your terminal, or log in to any of the department servers.

=== Creating a Directory ===
To create a new directory, you can use the <code>mkdir</code> command, which quite literally stands for "make directory." Simply type the command followed by the name of the directory you wish to create:<syntaxhighlight lang="linux-config">
mkdir MyDirectory
</syntaxhighlight>

=== Moving Between Directories ===
The change directory command is called <code>cd</code> and can be used to efficiently switch between directories. To move into our newly created directory, type:<syntaxhighlight lang="linux-config">
cd MyDirectory
</syntaxhighlight>If we want to go backwards, we can use <code>..</code>, which always means to go back one folder from where you currently are.<syntaxhighlight lang="linux-config">
cd ..
</syntaxhighlight>If we want to return to our root directory, we can simply type the command with nothing after it.<syntaxhighlight lang="linux-config">
cd
</syntaxhighlight>I can check my location using the present working directory command or <code>pwd</code>. Typing <code>pwd</code> into the terminal will return the path to your current working directory (i.e. where you are currently located in your file system) from the root.

=== Moving and Copying Files ===
Say I have a file in MyDirectory called myfile. I might want to move myfile into MySubDirectory. To do this, I would do as follows:<syntaxhighlight lang="linux-config">
mv myfile MySubDirectory/
</syntaxhighlight>The first thing after <code>mv</code> represents the file to move, and the second represents the destination. <code>mv</code> is also good for renaming files. I can change the name of myfile to mynewname as follows:<syntaxhighlight lang="linux-config">
mv myfile mynewname
</syntaxhighlight>This just renames the file in place without moving it elsewhere.

Sometimes, we don't want to move files but we do want to copy them. This is the <code>cp</code> command, and it works much the same way as the <code>mv</code> command. To use <code>cp</code>, you must specify the file or folder you wish to copy, and the destination to where you want to copy it.<syntaxhighlight lang="linux-config">
cp myfile ../
</syntaxhighlight>This creates a new copy of myfile in the previous directory on the path to my current working directory. If I have a directory, I might want to copy its entire contents to a new directory. This requires the <code>-r</code> flag, meaning recursive. This essentially means that for every element in the directory, we recursively copy those files to the new location. Flags can often be added to a command to provide additional information or for greater ease of use. <syntaxhighlight lang="linux-config">
cp -r MyDirectory ../
</syntaxhighlight>This command copies everything that exists in MyDirectory recursively and moves it back one directory from the current working directory.

=== Deleting Files ===
You must be extremely careful when deleting files through terminal. Unlike when you are using your general user interface and click and drag something to the trash, files cannot easily be restored. Never remove something without being absolutely certain it is what you intend to do and that there are no unintended consequences of its deletion.

The removal command is <code>rm</code>. Just like with <code>cp</code>, this can include the <code>-r</code> flag or not to be done recursively. <syntaxhighlight lang="linux-config">
rm myfile
</syntaxhighlight>This code simply deletes the file myfile.<syntaxhighlight lang="linux-config">
rm -r MyDirectory
</syntaxhighlight>This deletes everything that is in the directory MyDirectory recursively.

=== Determining What is in a Directory ===
The list command lists all of the files that are in your current working directory. <syntaxhighlight lang="linux-config">
ls
</syntaxhighlight>You can get more information about different files by using different flags. For instance, the flag <code>-l</code> gives additional information about your read/write access for a file, when it was last created/manipulated, who has access to it, the owner, etc. <syntaxhighlight lang="linux-config">
ls -l
</syntaxhighlight>Running just <code>ls</code> tells you what is in your current working directory, but you can provide a path to determine what is in another directory. The following lists everything that is in MyDirectory, where ~ just indicates the root:<syntaxhighlight lang="linux-config">
ls ~/MyDirectory/
</syntaxhighlight>

=== Creating and Editing a New File ===
You can create and edit new files directly in the terminal. There are many commands for doing this: this author's personal favorite is <code>vim</code>. <syntaxhighlight lang="linux-config">
vim my_python_script.py
</syntaxhighlight>This creates a file called my_python_script.py. When you run this line in the terminal, an editor will appear. To insert information into the editor, click <code>i</code>, then type whatever Python code you would like to execute. To stop editing the file, click the escape key and then type <code>:wq!</code> which will write your files you the file my_python_script.py. You delete entire lines by typing <code>dd</code>. These are just some of the many useful tools that <code>vim</code> offers. To edit a file once it has already been created, you need only type <code>vim my_python_script.py</code> once again.

You can view the contents of a file without using <code>vim</code> via <code>cat</code>.<syntaxhighlight lang="linux-config">
cat my_python_file.py
</syntaxhighlight>This line of code will show you everything that is in my_python_file.py without taking you to a terminal editor. This is great for viewing logs and output files from running chunks of code on the servers.

If your terminal becomes bogged down with too much text, simply type <code>clear</code> to reset the view.

=== Running a Python Script ===
This next part depends on if you have Python installed on your computer and what you have it installed as. Some might have it saved just as <code>python</code>, others might have <code>python3/python2</code>, depending on if you have multiple versions.

Say I have the following Python code in my_python_file.py, saved in my current working directory:<syntaxhighlight lang="python">
print("Hello, World!")
</syntaxhighlight>I can then do the following on the terminal:<syntaxhighlight lang="linux-config">
python my_python_file.py
</syntaxhighlight>The terminal will then show "Hello, World!"

I can also redirect the output of the script to another file if I wish to save it. This can be done via:<syntaxhighlight lang="linux-config">
python my_python_file.py > output.out
</syntaxhighlight>This saves whatever is printed to a file called output.out. If I ran <code>cat output.out</code> I would see "Hello, World!" printed.

If I want to run a Python script that may take some time, and I wish to walk away from my computer or do something else in the terminal while it is running, I can use the <code>nohup</code> command. This will allow the script to run in the background, while I do other things on my computer.<syntaxhighlight lang="linux-config">
nohup python my_python_file.py &
</syntaxhighlight>Adding the & allows me to continue working in my current terminal without waiting for the code to finish running.

=== Additional Resources ===

* [https://swcarpentry.github.io/shell-novice/ Software Carpentry]
* [https://www.learnshell.org/ Learn Shell]
* [https://mywiki.wooledge.org/BashGuide Bash Guide]

Main Page

2025-07-11T14:25:53Z

Ecucuzze: /* Computing Office Hours */

Anyone can read this wiki.  If you want to write/edit stuff in the wiki, you will need to log in with your account provided by the Statistics & Data Science Department computing group. This Wiki is intended to be a repository for instruction, information and ideas related to the many computing environments available to members and affiliates of the Statistics & Data Science Department.

== Getting to know the computing resources & getting support ==

If you're interested in knowing about what resources you have access to (and/or) what resources you have access to that might solve a problem you're having, please visit our [[What_resources_when|"what resources when" article]].

To get support that is beyond this website please attend office hours below and/or email [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu] for support (for more details about how the support ticketing works through [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu] + ServiceNow please see the [[Service_now|ServiceNow]] article).

Reset your StatDS password here - https://util.stat.cmu.edu/passwordreset/

== Department Contacts ==

• [mailto:events@stat.cmu.edu events@stat.cmu.edu]: seminars, conferences & workshops, and calendar updates

• [mailto:news@stat.cmu.edu news@stat.cmu.edu]: dept/dept member-related public news & announcements

• [mailto:lists@stat.cmu.edu lists@stat.cmu.edu]: department email lists

• [mailto:webmaster@stat.cmu.edu webmaster@stat.cmu.edu]: website updates

• [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu]: computer/tech requests

• [mailto:rooms@stat.cmu.edu rooms@stat.cmu.edu]: request space for meetings and TA office hours

• [mailto:reimbursements@stat.cmu.edu reimbursements@stat.cmu.edu]: travel and reimbursement requests

== Computing Office Hours ==

These office hour opportunities provide a dedicated time for you to come ask questions about research computing.  These include code advice, how to access certain resources, what the best resources are for your project, or to learn about new connections to help you progress in your education and research. Staff Office hour times will be sent out each week on Monday as a calendar invite; accept any of the invitations to add it to your personal calendar. PhD TA Office Hours are below, unless a different time is communicated.

#PhD TA Office Hours for the first half of summer 2025 are held by Elizabeth Cucuzzella in FMS Room 320 on Mondays from 2:30 pm to 4:30 pm. PhD students can also message on the #computing channel on slack.
#Staff Office Hours are held by appointment by Carl Skipper in BH 129G. Schedule via [mailto:lists@stat.cmu.edu email].

See [[Computing_office_hour_usage|computing office hour usage]] for what office hours can be used for.

== Computing Wiki Pages ==

=== Managing servers and environments ===

==== Virtual environments ====
*[[Virtual Environment Introduction|Why use virtual environments?]]
*R's [[Renv]]
*Python's [[Python_virtual_environments|virtual environments (2 options)]]
*Old instructions for [[Conda with Python (Reference Only)|conda with Python]]

==== Containers ====
*[[Introduction to containers]]
*[[Docker, and why-you-should-use-it-for-your-research]]
*[[Tutorials|Basic tutorial]]
*[[Running Docker locally and on Hydras]]

=== Department computing resources ===

==== Printing ====
*[[Department_Printers|Department Printers]]

==== Web Hosting ====
#[[How to host your webpage under stat.cmu.edu]]
#[[Bulding your website with Hugo]]

==== Servers ====
*[[Linux_and_department_servers|Department Servers]]. This wiki introduces you to the '''hydra''' servers, basic Linux commands, and more.
*#[[Terminal_multiplexers|Terminal multiplexers (screen and tmux)]]. This wiki presents tools for use on the ''hydra'' that can allow you to run jobs even when your not on the server, and much much more!
*#[[Python on the server]]. This wiki introduces uses command line approaches to running <code>python</code> on a virtual machine, and how to use jupyter notebooks through the server as well.
*#[[R on the server]]. This wiki introduces uses command line approaches to running <code>R</code> on a virtual machine

*[[Department GPU Servers]], for small jobs needing 1 GPU.
*[[GPU Cluster]], for larger jobs that may need multiple GPUs and many CPUs.
*[[PSC]], the Pittsburgh Supercomputing Center. Free accounts and usage credits are available
*#[[ACCESS]]
*#[[Slurm]]. This wiki introduces the job management system uses at PSC.
*#[[Parallelization_on_PSC/Bridges-2]]
*#[[Access PSC compute nodes directly from VSCode]]
*Azure
*#[[Azure_getting_started|Azure: getting started]]. This wiki walks you through the steps to create a virtual machine through Azure
*#[[How to obtain a virtual machine with a GPU]]
*#[[Attach a disk to a virtual machine]]

=== Development guides ===

====[[Parallel_computing|Parallelization]]====
*[[Parallelization_in_r|R-specific parallelization]]
*[[Parallelization_in_python|Python-specific parallelization]]

==== Package development ====
*[[Developing_R_packages|R package development]]
*[[Developing_python_packages|Python package development]]
[[Terminal automation]]. This wiki introduces approaches to automate Linux/OS command line calls.
*[[Bash files]]. This wiki presents potential uses of Bash files including automated file/folder creation, parallelization and much more!
*[[Make]]. This wiki presents potential uses of <code>make</code> and Makefiles including automated workflow and dependency tracking
*[[Jumping to frequently used folders and automating environment setups]]

==== Version Control ====
See the guide provided in the site of [https://36-750.github.io/tools/version-control/ 36750 - Statistical Computing]

==== Reference management ====
#Zotero
#Mendeley

==== Integrated Development Environments (IDEs) ====
#[[VSCode|VSCode]]
#PyCharm
#RStudio
#Eclipse

==== Text Editors ====
#Vim
#Nano
#Emacs
#Atom
#Sublime

See also the guide provided in the site of [https://36-750.github.io/tools/editors/ 36750 - Statistical Computing]

==== AI Tools ====
* [[ChatGPT and other LLMs]]

*[[github_copilot|GitHub Copilot]]

==== Miscellaneous====
#[[Windows_Subsystem_for_Linux_(WSL)_and_Python|Windows Subsystem for Linux (WSL) and Python]]
#[[Amphetamine|Amphetamine: How to keep your computer awake]]

===General Department Computing Resources===

*Fall 2021 [[Computing_seminar_resources|Computing Seminar Resources]] (video and slides)
*[[Basic Linux Commands|Introduction to Basic Linux Commands]]

==Ideas for the Wiki==

Please see the [[TODO|TODO page]] to post suggestions and errata.

==Using the Wiki==

*Here is some [[Generic_Wiki_Information|Generic Wiki Information]] that you may be interested in, such as a users' guide and other resources for Wiki users.

Command Line Basics

2025-07-10T19:44:44Z

Ecucuzze:

Command Line Basics

2025-07-10T19:33:40Z

Ecucuzze:

Main Page

2025-07-10T19:23:06Z

Ecucuzze: /* General Department Computing Resources */

Anyone can read this wiki.  If you want to write/edit stuff in the wiki, you will need to log in with your account provided by the Statistics & Data Science Department computing group. This Wiki is intended to be a repository for instruction, information and ideas related to the many computing environments available to members and affiliates of the Statistics & Data Science Department.

== Getting to know the computing resources & getting support ==

If you're interested in knowing about what resources you have access to (and/or) what resources you have access to that might solve a problem you're having, please visit our [[What_resources_when|"what resources when" article]].

To get support that is beyond this website please attend office hours below and/or email [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu] for support (for more details about how the support ticketing works through [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu] + ServiceNow please see the [[Service_now|ServiceNow]] article).

Reset your StatDS password here - https://util.stat.cmu.edu/passwordreset/

== Department Contacts ==

• [mailto:events@stat.cmu.edu events@stat.cmu.edu]: seminars, conferences & workshops, and calendar updates

• [mailto:news@stat.cmu.edu news@stat.cmu.edu]: dept/dept member-related public news & announcements

• [mailto:lists@stat.cmu.edu lists@stat.cmu.edu]: department email lists

• [mailto:webmaster@stat.cmu.edu webmaster@stat.cmu.edu]: website updates

• [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu]: computer/tech requests

• [mailto:rooms@stat.cmu.edu rooms@stat.cmu.edu]: request space for meetings and TA office hours

• [mailto:reimbursements@stat.cmu.edu reimbursements@stat.cmu.edu]: travel and reimbursement requests

== Computing Office Hours ==

These office hour opportunities provide a dedicated time for you to come ask questions about research computing.  These include code advice, how to access certain resources, what the best resources are for your project, or to learn about new connections to help you progress in your education and research. Staff Office hour times will be sent out each week on Monday as a calendar invite; accept any of the invitations to add it to your personal calendar. PhD TA Office Hours are below, unless a different time is communicated.

#PhD TA Office Hours for the first half of summer 2025 are held by Julia Walchessen in FMS Room 320 on Mondays from 2:30 pm to 4:30 pm
#Staff Office Hours are held by appointment by Carl Skipper in BH 129G. Schedule via [mailto:lists@stat.cmu.edu email].

See [[Computing_office_hour_usage|computing office hour usage]] for what office hours can be used for.

== Computing Wiki Pages ==

=== Managing servers and environments ===

==== Virtual environments ====
*[[Virtual Environment Introduction|Why use virtual environments?]]
*R's [[Renv]]
*Python's [[Python_virtual_environments|virtual environments (2 options)]]
*Old instructions for [[Conda with Python (Reference Only)|conda with Python]]

==== Containers ====
*[[Introduction to containers]]
*[[Docker, and why-you-should-use-it-for-your-research]]
*[[Tutorials|Basic tutorial]]
*[[Running Docker locally and on Hydras]]

=== Department computing resources ===

==== Printing ====
*[[Department_Printers|Department Printers]]

==== Web Hosting ====
#[[How to host your webpage under stat.cmu.edu]]
#[[Bulding your website with Hugo]]

==== Servers ====
*[[Linux_and_department_servers|Department Servers]]. This wiki introduces you to the '''hydra''' servers, basic Linux commands, and more.
*#[[Terminal_multiplexers|Terminal multiplexers (screen and tmux)]]. This wiki presents tools for use on the ''hydra'' that can allow you to run jobs even when your not on the server, and much much more!
*#[[Python on the server]]. This wiki introduces uses command line approaches to running <code>python</code> on a virtual machine, and how to use jupyter notebooks through the server as well.
*#[[R on the server]]. This wiki introduces uses command line approaches to running <code>R</code> on a virtual machine

*[[Department GPU Servers]], for small jobs needing 1 GPU.
*[[GPU Cluster]], for larger jobs that may need multiple GPUs and many CPUs.
*[[PSC]], the Pittsburgh Supercomputing Center. Free accounts and usage credits are available
*#[[ACCESS]]
*#[[Slurm]]. This wiki introduces the job management system uses at PSC.
*#[[Parallelization_on_PSC/Bridges-2]]
*#[[Access PSC compute nodes directly from VSCode]]
*Azure
*#[[Azure_getting_started|Azure: getting started]]. This wiki walks you through the steps to create a virtual machine through Azure
*#[[How to obtain a virtual machine with a GPU]]
*#[[Attach a disk to a virtual machine]]

=== Development guides ===

====[[Parallel_computing|Parallelization]]====
*[[Parallelization_in_r|R-specific parallelization]]
*[[Parallelization_in_python|Python-specific parallelization]]

==== Package development ====
*[[Developing_R_packages|R package development]]
*[[Developing_python_packages|Python package development]]
[[Terminal automation]]. This wiki introduces approaches to automate Linux/OS command line calls.
*[[Bash files]]. This wiki presents potential uses of Bash files including automated file/folder creation, parallelization and much more!
*[[Make]]. This wiki presents potential uses of <code>make</code> and Makefiles including automated workflow and dependency tracking
*[[Jumping to frequently used folders and automating environment setups]]

==== Version Control ====
See the guide provided in the site of [https://36-750.github.io/tools/version-control/ 36750 - Statistical Computing]

==== Reference management ====
#Zotero
#Mendeley

==== Integrated Development Environments (IDEs) ====
#[[VSCode|VSCode]]
#PyCharm
#RStudio
#Eclipse

==== Text Editors ====
#Vim
#Nano
#Emacs
#Atom
#Sublime

See also the guide provided in the site of [https://36-750.github.io/tools/editors/ 36750 - Statistical Computing]

==== AI Tools ====
* [[ChatGPT and other LLMs]]

*[[github_copilot|GitHub Copilot]]

==== Miscellaneous====
#[[Windows_Subsystem_for_Linux_(WSL)_and_Python|Windows Subsystem for Linux (WSL) and Python]]
#[[Amphetamine|Amphetamine: How to keep your computer awake]]

===General Department Computing Resources===

*Fall 2021 [[Computing_seminar_resources|Computing Seminar Resources]] (video and slides)
*[[Basic Linux Commands|Introduction to Basic Linux Commands]]

==Ideas for the Wiki==

Please see the [[TODO|TODO page]] to post suggestions and errata.

==Using the Wiki==

*Here is some [[Generic_Wiki_Information|Generic Wiki Information]] that you may be interested in, such as a users' guide and other resources for Wiki users.

Command Line Basics

2025-07-10T19:22:49Z

Ecucuzze:

Command Line Basics

2025-07-10T19:04:56Z

Ecucuzze: creating basic linux commands the right way

This page serves as an introduction to basic commands for managing your file system on Linux.

Index.php?title=BasicLinuxCommands

2025-07-10T19:03:03Z

Ecucuzze: Introduction to Basic Linux Commands

This is an introduction to basic linux commands page

Main Page

2025-07-10T17:53:24Z

Ecucuzze: /* Miscellanea */

Anyone can read this wiki.  If you want to write/edit stuff in the wiki, you will need to log in with your account provided by the Statistics & Data Science Department computing group. This Wiki is intended to be a repository for instruction, information and ideas related to the many computing environments available to members and affiliates of the Statistics & Data Science Department.

== Getting to know the computing resources & getting support ==

If you're interested in knowing about what resources you have access to (and/or) what resources you have access to that might solve a problem you're having, please visit our [[What_resources_when|"what resources when" article]].

To get support that is beyond this website please attend office hours below and/or email [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu] for support (for more details about how the support ticketing works through [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu] + ServiceNow please see the [[Service_now|ServiceNow]] article).

Reset your StatDS password here - https://util.stat.cmu.edu/passwordreset/

== Department Contacts ==

• [mailto:events@stat.cmu.edu events@stat.cmu.edu]: seminars, conferences & workshops, and calendar updates

• [mailto:news@stat.cmu.edu news@stat.cmu.edu]: dept/dept member-related public news & announcements

• [mailto:lists@stat.cmu.edu lists@stat.cmu.edu]: department email lists

• [mailto:webmaster@stat.cmu.edu webmaster@stat.cmu.edu]: website updates

• [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu]: computer/tech requests

• [mailto:rooms@stat.cmu.edu rooms@stat.cmu.edu]: request space for meetings and TA office hours

• [mailto:reimbursements@stat.cmu.edu reimbursements@stat.cmu.edu]: travel and reimbursement requests

== Computing Office Hours ==

These office hour opportunities provide a dedicated time for you to come ask questions about research computing.  These include code advice, how to access certain resources, what the best resources are for your project, or to learn about new connections to help you progress in your education and research. Staff Office hour times will be sent out each week on Monday as a calendar invite; accept any of the invitations to add it to your personal calendar. PhD TA Office Hours are below, unless a different time is communicated.

#PhD TA Office Hours for the first half of summer 2025 are held by Julia Walchessen in FMS Room 320 on Mondays from 2:30 pm to 4:30 pm
#Staff Office Hours are held by appointment by Carl Skipper in BH 129G. Schedule via [mailto:lists@stat.cmu.edu email].

See [[Computing_office_hour_usage|computing office hour usage]] for what office hours can be used for.

== Computing Wiki Pages ==

=== Managing servers and environments ===

==== Virtual environments ====
*[[Virtual Environment Introduction|Why use virtual environments?]]
*R's [[Renv]]
*Python's [[Python_virtual_environments|virtual environments (2 options)]]
*Old instructions for [[Conda with Python (Reference Only)|conda with Python]]

==== Containers ====
*[[Introduction to containers]]
*[[Docker, and why-you-should-use-it-for-your-research]]
*[[Tutorials|Basic tutorial]]
*[[Running Docker locally and on Hydras]]

=== Department computing resources ===

==== Printing ====
*[[Department_Printers|Department Printers]]

==== Web Hosting ====
#[[How to host your webpage under stat.cmu.edu]]
#[[Bulding your website with Hugo]]

==== Servers ====
*[[Linux_and_department_servers|Department Servers]]. This wiki introduces you to the '''hydra''' servers, basic Linux commands, and more.
*#[[Terminal_multiplexers|Terminal multiplexers (screen and tmux)]]. This wiki presents tools for use on the ''hydra'' that can allow you to run jobs even when your not on the server, and much much more!
*#[[Python on the server]]. This wiki introduces uses command line approaches to running <code>python</code> on a virtual machine, and how to use jupyter notebooks through the server as well.
*#[[R on the server]]. This wiki introduces uses command line approaches to running <code>R</code> on a virtual machine

*[[Department GPU Servers]], for small jobs needing 1 GPU.
*[[GPU Cluster]], for larger jobs that may need multiple GPUs and many CPUs.
*[[PSC]], the Pittsburgh Supercomputing Center. Free accounts and usage credits are available
*#[[ACCESS]]
*#[[Slurm]]. This wiki introduces the job management system uses at PSC.
*#[[Parallelization_on_PSC/Bridges-2]]
*#[[Access PSC compute nodes directly from VSCode]]
*Azure
*#[[Azure_getting_started|Azure: getting started]]. This wiki walks you through the steps to create a virtual machine through Azure
*#[[How to obtain a virtual machine with a GPU]]
*#[[Attach a disk to a virtual machine]]

=== Development guides ===

====[[Parallel_computing|Parallelization]]====
*[[Parallelization_in_r|R-specific parallelization]]
*[[Parallelization_in_python|Python-specific parallelization]]

==== Package development ====
*[[Developing_R_packages|R package development]]
*[[Developing_python_packages|Python package development]]
[[Terminal automation]]. This wiki introduces approaches to automate Linux/OS command line calls.
*[[Bash files]]. This wiki presents potential uses of Bash files including automated file/folder creation, parallelization and much more!
*[[Make]]. This wiki presents potential uses of <code>make</code> and Makefiles including automated workflow and dependency tracking
*[[Jumping to frequently used folders and automating environment setups]]

==== Version Control ====
See the guide provided in the site of [https://36-750.github.io/tools/version-control/ 36750 - Statistical Computing]

==== Reference management ====
#Zotero
#Mendeley

==== Integrated Development Environments (IDEs) ====
#[[VSCode|VSCode]]
#PyCharm
#RStudio
#Eclipse

==== Text Editors ====
#Vim
#Nano
#Emacs
#Atom
#Sublime

See also the guide provided in the site of [https://36-750.github.io/tools/editors/ 36750 - Statistical Computing]

==== AI Tools ====
* [[ChatGPT and other LLMs]]

*[[github_copilot|GitHub Copilot]]

==== Miscellaneous====
#[[Windows_Subsystem_for_Linux_(WSL)_and_Python|Windows Subsystem for Linux (WSL) and Python]]
#[[Amphetamine|Amphetamine: How to keep your computer awake]]

===General Department Computing Resources===

*Fall 2021 [[Computing_seminar_resources|Computing Seminar Resources]] (video and slides)

==Ideas for the Wiki==

Please see the [[TODO|TODO page]] to post suggestions and errata.

==Using the Wiki==

*Here is some [[Generic_Wiki_Information|Generic Wiki Information]] that you may be interested in, such as a users' guide and other resources for Wiki users.

Parallelization in r

2025-07-01T14:19:02Z

Ecucuzze:

There are a few ways to parallelize within <code>R</code>, generally broken up into 3 different approaches the capture different “for-loop” frameworks in <code>R</code>. They include:

#Parallelization in straight <code>for</code>-loop-style with <code>foeach</code>
#Parallelization in <code>apply</code>-style loops with <code>mclapply</code>
#Parallelization with <code>tidyverse</code> nesting-style loops with <code>multidplyr</code>

The final option in the above list is still too experimental and limited to truly encourage.

== General Starting Point ==

To start parallelizing within <code>R</code> , we need to have some understanding of how many cores we have. <syntaxhighlight lang="r">library(parallel)
numCores <- detectCores()
numCores</syntaxhighlight>

This will tell you the total number of cores on your computer. We often recommend using less than the total number of cores when working on your personal computer or on the department servers to leave room for other jobs. You can always determine the current server system use by checking the hydra usage page.

= for-loop style: <code>foreach</code> and <code>doParallel</code> library (<code>foreach () %dopar%</code>) =

The combination of <code>foreach</code> and <code>doParallel</code> can convert for-loop style coding into parallel processing and provide a lot of additional parameters that can help reduce unnecessary memory usage, provide progress reports, and more.

In the following presentation, we’ll demonstrate the speed obtained with different approaches using a simulation function defined below:<syntaxhighlight lang="r">my_k <- rep(1:14, each = 20)

my_simulation <- function(k) {
x <- runif(30,-3, 3)
y <- 2*x + 2* rnorm(n = 30)
x0 <- runif(10000,-3, 3)
y0 <- 2*x0 + 2*rnorm(n = 10000)

my_lm <- lm(y ~ poly(x,k))
y0_hat <- predict(my_lm, data.frame(x = x0))
return(mean((y0-y0_hat)^2))
}</syntaxhighlight>

== Single Core Analysis ==

We naturally want to compare a parallelization approach with the base, single core analysis, which can be done using a standard for loop over various values of <code>k</code> :<syntaxhighlight lang="r">#base for-loop
system.time({
storage_f <- c()
for (k in my_k) {
storage_f <- c(storage_f, my_simulation(k))
}
})</syntaxhighlight>
<syntaxhighlight lang="r">## user system elapsed
## 1.811 0.633 2.449</syntaxhighlight>

In the <code>foreach</code> package, we can also use just a single core in the following way which results in approximately the same run time as the previous code segment:<syntaxhighlight lang="r">#single-core `foreach`
library(foreach)
system.time({
storage_fe <- foreach (k=my_k, .combine=c) %do% {
my_simulation(k)
}
})</syntaxhighlight>
<syntaxhighlight lang="r">## user system elapsed
## 1.876 0.640 2.522</syntaxhighlight>

It should be clear that the <code>foreach</code> function works very similar to the base <code>for</code>-loop; the difference in the syntax is mainly the fact that there is a <code>%do%</code> before the opening bracket. Additionally, notice that the loops of <code>k</code> for values in <code>my_k</code> is expressed differently than in a standard <code>for</code> loop (<code>k in my_k</code> vs <code>k = my_k</code>).

== Multi-Core Analysis ==

To use multiple cores with <code>foreach</code> we just change the above code to:<syntaxhighlight lang="r">library(doParallel)
registerDoParallel(numCores-2) # use multicore, set to the number of our cores

system.time({
storage_p <- foreach (k=my_k, .combine=c) %dopar% {
my_simulation(k)
}
})
stopImplicitCluster()</syntaxhighlight>
<syntaxhighlight lang="r">## user system elapsed
## 3.513 2.520 0.601</syntaxhighlight>

The last number under “elapsed” suggest that although we used more core + user time overall, we completed the overall process faster than without parallelization. The <code>%dopar%</code> represents "do parallelization."

== Additional Parameters ==

#<code>.combine</code>: this parameter tells the system how you’d like to return each step of the for-loop when combined at the end. Above we used <code>.combine = c</code> to get a vector back, but we could also use things like <code>rbind</code>, <code>cbind</code>. Note that the default is a list.
#Memory and library passing: Sometimes you have a lot of functions or memory in your RAM before your run the parallelized for loop. The <code>foreach</code> function tries to reduce the amount memory used for each parallel process by constraining which functions are passed in. To specify which packages you’d like to use, use the <code>.packages</code> command in <code>foreach.</code> For example: <code>.packages = c("dplyr", "tidyr")</code>. Additionally, you can constrain what data is passed into each process with the parameter <code>.export</code> which uses strings relative to your global environment.
#Progress bars: To have a running progress bar, one can use the package <code>doSNOW</code>. For the previous process we can add a progress bar with the following set of code: <syntaxhighlight lang="r">library(doSNOW)
cl <- makeCluster(numCores-4)
registerDoSNOW(cl)
pb <- txtProgressBar(max = length(my_k), style = 3)
progress <- function(n) setTxtProgressBar(pb, n)
opts <- list(progress = progress)

storage_p <- foreach (k=my_k, .combine=c,
.options.snow = opts) %dopar% {
my_simulation(k)
}
parallel::stopCluster(cl)</syntaxhighlight>
<syntaxhighlight lang="r">|============================================================== | 83%</syntaxhighlight>

 

= lapply style: <code>parallel</code> library (<code>mclapply</code>) =

The second package we will look at is <code>parallel</code>, which includes the function <code>mclapply</code> which can be used very similar to an <code>lapply</code>.

== Single Core Analysis ==

For a single-core analysis of the same simulation seen in the previous section, we could run:<syntaxhighlight lang="r">
system.time({
results <- lapply(my_k, my_simulation)
my_data <- data.frame(k = my_k,
values = unlist(results))
})</syntaxhighlight>
<syntaxhighlight lang="r">## user system elapsed
## 1.819 0.641 2.467</syntaxhighlight>

== Multi-Core Analysis ==

Now, with the <code>mclapply</code> function we can do basically the same thing as before. The difference here is the same as the difference between using <code>for</code> and <code>lapply</code> in base <code>R</code>. <code>mclapply</code> applies each of the values in <code>my_k</code> to the simulation, and parallelizes it across <code>numCores</code> cores.<syntaxhighlight lang="r">system.time({
results <- mclapply(my_k, my_simulation, mc.cores = numCores)
my_data <- data.frame(k = my_k,
values = unlist(results))
})</syntaxhighlight>
<syntaxhighlight lang="r">## user system elapsed
## 2.208 1.636 0.522</syntaxhighlight>

== Extra Resources: ==

*[https://nceas.github.io/oss-lessons/parallel-computing-in-r/parallel-computing-in-r.html Matt Jones]
*[https://blog.dominodatalab.com/multicore-data-science-r-python/ R and python more options]
*<code>parallel</code> [https://stat.ethz.ch/R-manual/R-devel/library/parallel/doc/parallel.pdf manual]
*<code>doParallel</code> + <code>foreach</code> [https://cran.r-project.org/web/packages/doParallel/vignettes/gettingstartedParallel.pdf vignette]
= The Fastest One as of February 2024: <code>BiocParallel</code>=

All of the above are good solutions, but from practical experience, we have seen that sometimes the previous methods can incur high overheads. A consistently faster solution comes from a package developed and maintained by the R bioinformatics community:  <code>BioConductor</code>. The package is named <code>BiocParallel </code>and its interface is almost exactly the same as the one for the <code>parallel</code> package seen in the previous section. The only difference is that we need another package.

First of all, install the <code>BiocParallel</code> package manager via
<pre tabindex="0">install.packages("BiocManager")
</pre>

This will make available an <code>install </code>command, which we use as follows:
<pre>install("BiocParallel")</pre>

We can then use a simple command. Assuming the function to run multiple times in parallel is named <code>foo</code>, we type:
<pre tabindex="0">library('BiocParallel')
system.time(
y <- bplapply(seq_len(100), foo)
)</pre>

See this [https://lcolladotor.github.io/2016/03/07/biocparallel/ article] for more information.

Parallelization in r

2025-07-01T14:18:32Z

Ecucuzze: Updated and cleaned up the language on this page.

There are a few ways to parallelize within <code>R</code>, generally broken up into 3 different approaches the capture different “for-loop” frameworks in <code>R</code>. They include:

#Parallelization in straight <code>for</code>-loop-style with <code>foeach</code>
#Parallelization in <code>apply</code>-style loops with <code>mclapply</code>
#Parallelization with <code>tidyverse</code> nesting-style loops with <code>multidplyr</code>

The final option in the above list is still too experimental and limited to truly encourage.

== General Starting Point ==

To start parallelizing within <code>R</code> , we need to have some understanding of how many cores we have. <syntaxhighlight lang="r">library(parallel)
numCores <- detectCores()
numCores</syntaxhighlight>

This will tell you the total number of cores on your computer. We often recommend using less than the total number of cores when working on your personal computer or on the department servers to leave room for other jobs. You can always determine the current server system use by checking the hydra usage page.

= for-loop style: <code>foreach</code> and <code>doParallel</code> library (<code>foreach () %dopar%</code>) =

The combination of <code>foreach</code> and <code>doParallel</code> can convert for-loop style coding into parallel processing and provide a lot of additional parameters that can help reduce unnecessary memory usage, provide progress reports, and more.

In the following presentation, we’ll demonstrate the speed obtained with different approaches using a simulation function defined below:<syntaxhighlight lang="r">my_k <- rep(1:14, each = 20)

my_simulation <- function(k) {
x <- runif(30,-3, 3)
y <- 2*x + 2* rnorm(n = 30)
x0 <- runif(10000,-3, 3)
y0 <- 2*x0 + 2*rnorm(n = 10000)

my_lm <- lm(y ~ poly(x,k))
y0_hat <- predict(my_lm, data.frame(x = x0))
return(mean((y0-y0_hat)^2))
}</syntaxhighlight>

== Single Core Analysis ==

We naturally want to compare a parallelization approach with the base, single core analysis, which can be done using a standard for loop over various values of <code>k</code> :<syntaxhighlight lang="r">#base for-loop
system.time({
storage_f <- c()
for (k in my_k) {
storage_f <- c(storage_f, my_simulation(k))
}
})</syntaxhighlight>
<syntaxhighlight lang="r">## user system elapsed
## 1.811 0.633 2.449</syntaxhighlight>

In the <code>foreach</code> package, we can also use just a single core in the following way which results in approximately the same run time as the previous code segment:<syntaxhighlight lang="r">#single-core `foreach`
library(foreach)
system.time({
storage_fe <- foreach (k=my_k, .combine=c) %do% {
my_simulation(k)
}
})</syntaxhighlight>
<syntaxhighlight lang="r">## user system elapsed
## 1.876 0.640 2.522</syntaxhighlight>

It should be clear that the <code>foreach</code> function works very similar to the base <code>for</code>-loop; the difference in the syntax is mainly the fact that there is a <code>%do%</code> before the opening bracket. Additionally, notice that the loops of <code>k</code> for values in <code>my_k</code> is expressed differently than in a standard <code>for</code> loop (<code>k in my_k</code> vs <code>k = my_k</code>).

== Multi-Core Analysis ==

To use multiple cores with <code>foreach</code> we just change the above code to:<syntaxhighlight lang="r">library(doParallel)
registerDoParallel(numCores-2) # use multicore, set to the number of our cores

system.time({
storage_p <- foreach (k=my_k, .combine=c) %dopar% {
my_simulation(k)
}
})
stopImplicitCluster()</syntaxhighlight>
<syntaxhighlight lang="r">## user system elapsed
## 3.513 2.520 0.601</syntaxhighlight>

The last number under “elapsed” suggest that although we used more core + user time overall, we completed the overall process faster than without parallelization. The <code>%dopar%</code> represents "do parallelization."

== Additional Parameters ==

#<code>.combine</code>: this parameter tells the system how you’d like to return each step of the for-loop when combined at the end. Above we used <code>.combine = c</code> to get a vector back, but we could also use things like <code>rbind</code>, <code>cbind</code>. Note that the default is a list.
#Memory and library passing: Sometimes you have a lot of functions or memory in your RAM before your run the parallelized for loop. The <code>foreach</code> function tries to reduce the amount memory used for each parallel process by constraining which functions are passed in. To specify which packages you’d like to use, use the <code>.packages</code> command in <code>foreach.</code> For example: <code>.packages = c("dplyr", "tidyr")</code>. Additionally, you can constrain what data is passed into each process with the parameter <code>.export</code> which uses strings relative to your global environment.
#Progress bars: To have a running progress bar, one can use the package <code>doSNOW</code>. For the previous process we can add a progress bar with the following set of code: <syntaxhighlight lang="r">library(doSNOW)
cl <- makeCluster(numCores-4)
registerDoSNOW(cl)
pb <- txtProgressBar(max = length(my_k), style = 3)
progress <- function(n) setTxtProgressBar(pb, n)
opts <- list(progress = progress)

storage_p <- foreach (k=my_k, .combine=c,
.options.snow = opts) %dopar% {
my_simulation(k)
}
parallel::stopCluster(cl)</syntaxhighlight>
<syntaxhighlight lang="r">|============================================================== | 83%</syntaxhighlight>

 

= lapply style: <code>parallel</code> library (<code>mclapply</code>) =

The second package we will look at is <code>parallel</code>, which includes the function <code>mclapply</code> which can be used very similar to an <code>lapply</code>.

== Single Core Analysis ==

For a single-core analysis of the same simulation seen in the previous section, we could run:<syntaxhighlight lang="r">
system.time({
results <- lapply(my_k, my_simulation)
my_data <- data.frame(k = my_k,
values = unlist(results))
})</syntaxhighlight>
<syntaxhighlight lang="r">## user system elapsed
## 1.819 0.641 2.467</syntaxhighlight>

== Multi-Core Analysis ==

Now, with the <code>mclapply</code> function we can do basically the same thing as before. The difference here is the same as the difference between using <code>for</code> and <code>lapply</code> in base <code>R</code>. <code>mclapply</code> applies each of the values in <code>my_k</code> to the simulation, and parallelizes it across <code>numCores</code> cores.<syntaxhighlight lang="r">system.time({
results <- mclapply(my_k, my_simulation, mc.cores = numCores)
my_data <- data.frame(k = my_k,
values = unlist(results))
})</syntaxhighlight>
<syntaxhighlight lang="r">## user system elapsed
## 2.208 1.636 0.522</syntaxhighlight>

== Extra Resources: ==

*[https://nceas.github.io/oss-lessons/parallel-computing-in-r/parallel-computing-in-r.html Matt Jones]
*[https://blog.dominodatalab.com/multicore-data-science-r-python/ R and python more options]
*<code>parallel</code> [https://stat.ethz.ch/R-manual/R-devel/library/parallel/doc/parallel.pdf manual]
*<code>doParallel</code> + <code>foreach</code> [https://cran.r-project.org/web/packages/doParallel/vignettes/gettingstartedParallel.pdf vignette]
== The Fastest One as of February 2024: <code>BiocParallel</code>==

All of the above are good solutions, but from practical experience, we have seen that sometimes the previous methods can incur high overheads. A consistently faster solution comes from a package developed and maintained by the R bioinformatics community:  <code>BioConductor</code>. The package is named <code>BiocParallel </code>and its interface is almost exactly the same as the one for the <code>parallel</code> package seen in the previous section. The only difference is that we need another package.

First of all, install the <code>BiocParallel</code> package manager via
<pre tabindex="0">install.packages("BiocManager")
</pre>

This will make available an <code>install </code>command, which we use as follows:
<pre>install("BiocParallel")</pre>

We can then use a simple command. Assuming the function to run multiple times in parallel is named <code>foo</code>, we type:
<pre tabindex="0">library('BiocParallel')
system.time(
y <- bplapply(seq_len(100), foo)
)</pre>

See this [https://lcolladotor.github.io/2016/03/07/biocparallel/ article] for more information.

Parallelization in python

2025-07-01T14:01:29Z

Ecucuzze:

In Python, parallelization is a bit more of a “wild west” in terms of the number of parallelization packages available. This wiki will attempt to present minor examples of some common approaches, along with an outline of things to think about in selecting the library you end up using. In general, the process in Python involves:

# defining a function that captures what you want to have parallelized
# calling that function repeatedly in the the parallelization package way (which often looks like a list-recursive call).

There are a few different general frameworks to accomplish this.

# '''General wrappers''' define the cores you’ll be using and then wrap that around the function call in a list-recursive way.
# '''Just in time compilers''', in addition to providing parallelization tools, provide decorators (e.g. <code>@jit</code>) for functions that allow for just in time compilation of the Python function into lower level machine code (like <code>C++</code>).

We will also discuss Python packages that are built specifically to parallelize jobs inside '''Jupyter notebooks'''.

= Non-parallelized: Single Core Example =

We naturally want to compare a parallelization approach with the base single core analysis, which can be done with the code as follows:

<syntaxhighlight lang="python">#base for-loop
import numpy as np
from sklearn.linear_model import LinearRegression

my_k = list(np.arange(14)+1)*20

def my_simulation(k):
"""
estamine the polynomial fit of degree k's MSE for a linear
relationship

Parameters
----------
k : int
degree of polynomial fit

Returns
-------
mse of fit based on 30 points applied to a test set of 10000 points
"""
x = np.random.uniform(size=30,low=-3,high=3)
y = 2*x + 2* np.random.normal(size=30)
y = y.reshape(-1,1)
X_poly = np.concatenate([x.reshape(-1,1)**kk
for kk in np.arange(1,k+1,dtype = int)],
axis = 1)

x0 = np.random.uniform(size=10000,low=-3,high=3)
y0 = 2*x0 + 2*np.random.normal(size=10000)
y0 = y0.reshape(-1,1)
X0_poly = np.concatenate([x0.reshape(-1,1)**kk
for kk in np.arange(1,k+1,dtype = int)],
axis = 1)

my_lm = LinearRegression().fit(X_poly,y)
y0_hat = my_lm.predict(X0_poly)

return(np.mean((y0-y0_hat)**2))
</syntaxhighlight>
<syntaxhighlight lang="python">import time

starttime = time.time()
[my_simulation(k) for k in my_k]
print(str(time.time() - starttime) + " seconds")</syntaxhighlight>
<syntaxhighlight lang="python">0.9454693794250488 seconds</syntaxhighlight>
= General Python Parallelization Packages =

As previously mentioned, there are a lot of packages for general parallelization. Two commonly used ones are <code>joblib</code> and <code>multiprocessing</code>. <code>joblib</code> looks a bit more like list-recursion and <code>multiprocessing</code> uses functions like <code>.map()</code>.

== <code>multiprocessing</code>/<code>multiprocess</code> ==

The <code>multiprocessing</code> package is actually built into Python, and works in a pretty straightforward manner. It also allows for synchronous vs. asynchronous parallelization, where synchronous means the order of the jobs must be followed, one after the next, and asynchronous means multiple tasks can be executed in small parts in overlapping periods. Note that many statistical uses of parallelization do not need synchronous parallelization.

Since both approaches have the same syntax we’ll be using <code>multiprocess</code> (as it works with <code>ipython</code> and <code>jupyter notebooks</code>). <code>multiprocessing</code> can have problems with <code>ipython</code> and <code>jupyter notebook</code>s. In <code>jupyter notebook</code>s, if you want to use <code>multiprocessing</code> , it needs to be wrapped in <code>if name == "main"</code> to properly fork processes; you can also save your process is as a .py file and import it into your <code>jupyter notebook.</code>

=== Implementation ===

First, you need to identify the number of cores you have:

<syntaxhighlight lang="python">import multiprocess as mp
#Could also using import multiprocessing as mp; the syntax is the same

num_cores = mp.cpu_count()</syntaxhighlight>
==== Synchronous ====

<syntaxhighlight lang="python">starttime = time.time()
with mp.Pool(processes=num_cores-2) as p:
results = p.map(my_simulation, my_k)
print(str(time.time() - starttime) + " seconds")</syntaxhighlight>
<syntaxhighlight lang="python">0.7638578414916992 seconds</syntaxhighlight>
==== Asynchronous ====

<syntaxhighlight lang="python">starttime = time.time()
with mp.Pool(processes=num_cores-2) as p:
results = p.map_async(my_simulation, my_k).get()
print(str(time.time() - starttime) + " seconds")
</syntaxhighlight>
<syntaxhighlight lang="python">0.8183681964874268 seconds</syntaxhighlight>
== <code>joblib</code> ==

The <code>joblib</code> package is pretty straightforward and requires you to structure things in terms of list-recursion with a few wrappers to help the process work as desired. Just like with <code>multiprocessing</code>, the number of cores needs to be computed before proceeding.

=== Implementation ===

<syntaxhighlight lang="python">import joblib

num_cores = joblib.cpu_count()</syntaxhighlight>To use <code>joblib</code> functionality, we can import the functions <code>Parallel</code> and <code>delayed</code>. <code>Parallel</code> creates an instance for parallelization with a set number of jobs based on the core availability. <code>delayed</code> creates deferred function calls to distribute computations across multiple cores; this is the function that allows for simultaneous task execution.

<syntaxhighlight lang="python">from joblib import Parallel, delayed</syntaxhighlight>
We then can move our list-recursive example into our <code>joblib</code> structure. We wrap <code>Parallel(...)()</code> around the inside of the list-recursion and <code>delayed()</code> around the functions that will be repeated multiple times.

<syntaxhighlight lang="python">starttime = time.time()
r = Parallel(n_jobs=num_cores-2)(delayed(my_simulation)(k) for k in my_k)
print(str(time.time() - starttime) + " seconds")</syntaxhighlight>
<syntaxhighlight lang="python">0.5306367874145508 seconds</syntaxhighlight>
=== Add-ons ===

There are multiple add-ons that can be desirable

<ol style="list-style-type: decimal;">
<li>'''Verbosity''' can be added with a single parameter use (as seen below). This will provide more detailed output information.
<syntaxhighlight lang="python">r = Parallel(n_jobs=num_cores-2, verbose=10)(delayed(my_simulation)(k) for k in my_k)</syntaxhighlight></li>
<li>'''Multiple parameters''' can be passed with redefinitions of functions to take groupings. To replace multiple for-loops, often the function <code>itertools.product()</code> is used for the parameters.</li></ol>

== <code>ray</code> ==
The package <code>ray</code> is an open source project for parallel and distributed computing in Python. <code>ray</code> is often preferred over <code>multiprocess/multiprocessing</code> because of its efficient machine error handling methodology and its ability to handle larger data types. <code>ray</code> allows for aggregation of results/values, meaning the output of one task can be used as input for subsequent tasks seamlessly. The implementation for <code>ray</code> is not dissimilar from that of <code>multiprocess</code>.

=== Implementation ===
<syntaxhighlight lang="python">import ray
import os</syntaxhighlight>In order to parallelize computation with <code>ray</code>, you must first initialize an instance of <code>ray</code>. Just like with <code>multiprocess</code>, the instance relies on the number of CPUs available.<syntaxhighlight lang="python">
ray.init(num_cpus = os.cpu_count())
</syntaxhighlight>Additionally, in order for <code>ray</code> to work, the function <code>my_simulation</code> needs to have the decorator <code>@ray.remote</code> prior to its heading. <syntaxhighlight lang="python">
@ray.remote
def my_simulation(k):
"""the rest of the function code goes here"""
</syntaxhighlight>Once we have initialized an instance based on our number of available CPUs, and we have made sure our function has the proper decorator, we can parallelize computation as follows. The <code>.remote()</code> passes the variable into the function, even if the previous computation has not finished running, and the <code>.get()</code> compiles the results once finished.<syntaxhighlight lang="python">
starttime = time.time()
futures = [my_simulation.remote(k) for k in my_k]
results = ray.get(futures)
print(str(time.time() - starttime) + " seconds")
</syntaxhighlight>This <code>.remote()</code> piece is useful because you can pass previous computations into it in order to have a parallelized, autoregressive work flow.<syntaxhighlight>
0.13494515419006348 seconds
</syntaxhighlight>

= Jupyter Notebook: <code>ipyparallel</code>=

For those that heavily use Jupyter notebooks, the package <code>ipyparallel</code> works quite nicely. Some benefits of this package include its verbosity, its ability to serialize numpy arrays cleanly, and the fact that it can distribute across multiple machines. This package also works with <code>numba.jit</code> and <code>cython</code>.

Before opening your <code>jupyter notebook</code> you’ll want to make sure the extension is enabled - which can be done from the command line with the following command or inside the <code>jupyter notebook</code> extensions tab.

<syntaxhighlight lang="bash">ipcluster nbextension enable</syntaxhighlight>
You’ll then need to go to the “IPython Clusters” tab in <code>jupyter noteook</code> and start some parallel clusters.

==<code>ipyparallel</code> parallelization ==

<syntaxhighlight lang="python">import ipyparallel as ipp</syntaxhighlight>
For direct execution we define <code>dv</code> and provide it with what it needs.

<syntaxhighlight lang="python">rc = ipp.Client()
dv = rc[:] # direct view</syntaxhighlight>
=== Passing Things to Each Cluster ===

<code>ipyparallel</code> provides ways for the user to pass only the necessary items to the the clusters as to not overload them.

<ol style="list-style-type: decimal;">
<li>Passing packages all together
<syntaxhighlight lang="python">with dv.sync_imports():
from sklearn.linear_model import LinearRegression
# getting dependencies on the clusters</syntaxhighlight></li>
<li>Passing variables:
Below is just an example - we don’t need to pass <code>my_k</code> since <code>dv</code> will directly get it when we want to use it.
<syntaxhighlight lang="python">dv["my_k"] = my_k</syntaxhighlight></li>
<li>We can also use the <code>%px</code> magick to run lines of code on all servers.
<syntaxhighlight lang="python">%px import numpy as np</syntaxhighlight></li></ol>

=== Actual Functionality ===

The actual functionality is very similar to <code>multiprocess</code>/<code>multiprocessing</code> with <code>maps</code> and <code>apply</code>.

==== Synchronous ====

<syntaxhighlight lang="python">%%timeit
results = dv.map_sync(my_simulation, my_k)</syntaxhighlight>
<syntaxhighlight lang="python">201 ms ± 34.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)</syntaxhighlight>
==== Asynchronous ====
For asynchronous, you need to use <code>.get()</code> in order to get the results at a later time/when they come in, since the work is not done in order and is instead chunked for efficiency.<syntaxhighlight lang="python">%%timeit
results = dv.map_async(my_simulation, my_k).get() </syntaxhighlight>
<syntaxhighlight lang="python">209 ms ± 28.1 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)</syntaxhighlight>
= Just in Time Compilers + Parallelization =

Just-in-time compilation is the compiling of your computer code during execution instead of beforehand. This means bytecode is not translated to machine code until run time, and is then executed directly. Code is continuously analyzed to determine where compilation/recompilation would speed up performance. Used in conjunction with parallelization, this is an efficient way to improve computational speed.

== <code>numba</code> ==
<code>numba</code> is a high performance Python compiler that translates Python/numpy code into fast, efficient machine code. Python is traditionally regarded as a "slow" language; using <code>numba</code>, you can approach the speeds of languages such as C. <code>numba</code> is specifically designed to work well with numpy arrays.

=== Implementation ===
Similar to <code>ray</code>, <code>numba</code> requires a decorator <code>@njit (parallel = True)</code> before the function. We will define a different function to be parallelized here to demonstrate <code>numba</code> <code>prange</code> which allows for efficient parallelization across loops.<syntaxhighlight lang="python">
A = np.random.rand(3, 3)
B = np.random.rand(3, 3)

@njit(parallel = True)
def my_simulation(A, B):
C = np.empty_like(A)
for i in prange(A.shape[0]):
for j in range(A.shape[1]):
C[i, j] = A[i, j] + B[i, j]
return C
</syntaxhighlight><code>prange</code> should not be nested and should only ever be used in the outer loop. This is a very simple example, but for computationally complex looping problems, <code>numba</code> is highly efficient. However, you need to ensure that you are not creating or declaring a variable within a loop that cannot be accessed outside of the loop, or else parallelization will not work.

= Additional Topics and Resources =

# [https://people.duke.edu/~ccc14/sta-663-2018/notebooks/S14A_Parallel_Programming_Introduction.html Duke’s statistical computing course] provides a set of examples for multiple Python parallelization packages

Parallelization in python

2025-06-30T18:30:57Z

Ecucuzze: /* Just in Time compilers + parallelization */

'''This article is still being developed - released earlier to align with Department seminar.'''

In Python, parallelization is a bit more of a “wild west” in terms of the number of parallelization packages available. This wiki will attempt to present minor examples of some common approaches, along with an outline of things to think about in selecting the library you end up using. In general, the process in Python involves:

# defining a function that captures what you want to have parallelized
# calling that function repeatedly in the the parallelization package way (which often looks like a list-recursive call).

There are a few different general frameworks to accomplish this.

# '''General wrappers''' define the cores you’ll be using and then wrap that around the function call in a list-recursive way.
# '''Just in time compilers''', in addition to providing parallelization tools, provide decorators (e.g. <code>@jit</code>) for functions that allow for just in time compilation of the Python function into lower level machine code (like <code>C++</code>).

We will also discuss Python packages that are built specifically to parallelize jobs inside '''Jupyter notebooks'''.

= Non-parallelized: Single Core Example =

We naturally want to compare a parallelization approach with the base single core analysis, which can be done with the code as follows:

<syntaxhighlight lang="python">#base for-loop
import numpy as np
from sklearn.linear_model import LinearRegression

my_k = list(np.arange(14)+1)*20

def my_simulation(k):
"""
estamine the polynomial fit of degree k's MSE for a linear
relationship

Parameters
----------
k : int
degree of polynomial fit

Returns
-------
mse of fit based on 30 points applied to a test set of 10000 points
"""
x = np.random.uniform(size=30,low=-3,high=3)
y = 2*x + 2* np.random.normal(size=30)
y = y.reshape(-1,1)
X_poly = np.concatenate([x.reshape(-1,1)**kk
for kk in np.arange(1,k+1,dtype = int)],
axis = 1)

x0 = np.random.uniform(size=10000,low=-3,high=3)
y0 = 2*x0 + 2*np.random.normal(size=10000)
y0 = y0.reshape(-1,1)
X0_poly = np.concatenate([x0.reshape(-1,1)**kk
for kk in np.arange(1,k+1,dtype = int)],
axis = 1)

my_lm = LinearRegression().fit(X_poly,y)
y0_hat = my_lm.predict(X0_poly)

return(np.mean((y0-y0_hat)**2))
</syntaxhighlight>
<syntaxhighlight lang="python">import time

starttime = time.time()
[my_simulation(k) for k in my_k]
print(str(time.time() - starttime) + " seconds")</syntaxhighlight>
<syntaxhighlight lang="python">0.9454693794250488 seconds</syntaxhighlight>
= General Python Parallelization Packages =

As previously mentioned, there are a lot of packages for general parallelization. Two commonly used ones are <code>joblib</code> and <code>multiprocessing</code>. <code>joblib</code> looks a bit more like list-recursion and <code>multiprocessing</code> uses functions like <code>.map()</code>.

== <code>multiprocessing</code>/<code>multiprocess</code> ==

The <code>multiprocessing</code> package is actually built into Python, and works in a pretty straightforward manner. It also allows for synchronous vs. asynchronous parallelization, where synchronous means the order of the jobs must be followed, one after the next, and asynchronous means multiple tasks can be executed in small parts in overlapping periods. Note that many statistical uses of parallelization do not need synchronous parallelization.

Since both approaches have the same syntax we’ll be using <code>multiprocess</code> (as it works with <code>ipython</code> and <code>jupyter notebooks</code>). <code>multiprocessing</code> can have problems with <code>ipython</code> and <code>jupyter notebook</code>s. In <code>jupyter notebook</code>s, if you want to use <code>multiprocessing</code> , it needs to be wrapped in <code>if name == "main"</code> to properly fork processes; you can also save your process is as a .py file and import it into your <code>jupyter notebook.</code>

=== Implementation ===

First, you need to identify the number of cores you have:

<syntaxhighlight lang="python">import multiprocess as mp
#Could also using import multiprocessing as mp; the syntax is the same

num_cores = mp.cpu_count()</syntaxhighlight>
==== Synchronous ====

<syntaxhighlight lang="python">starttime = time.time()
with mp.Pool(processes=num_cores-2) as p:
results = p.map(my_simulation, my_k)
print(str(time.time() - starttime) + " seconds")</syntaxhighlight>
<syntaxhighlight lang="python">0.7638578414916992 seconds</syntaxhighlight>
==== Asynchronous ====

<syntaxhighlight lang="python">starttime = time.time()
with mp.Pool(processes=num_cores-2) as p:
results = p.map_async(my_simulation, my_k).get()
print(str(time.time() - starttime) + " seconds")
</syntaxhighlight>
<syntaxhighlight lang="python">0.8183681964874268 seconds</syntaxhighlight>
== <code>joblib</code> ==

The <code>joblib</code> package is pretty straightforward and requires you to structure things in terms of list-recursion with a few wrappers to help the process work as desired. Just like with <code>multiprocessing</code>, the number of cores needs to be computed before proceeding.

=== Implementation ===

<syntaxhighlight lang="python">import joblib

num_cores = joblib.cpu_count()</syntaxhighlight>To use <code>joblib</code> functionality, we can import the functions <code>Parallel</code> and <code>delayed</code>. <code>Parallel</code> creates an instance for parallelization with a set number of jobs based on the core availability. <code>delayed</code> creates deferred function calls to distribute computations across multiple cores; this is the function that allows for simultaneous task execution.

<syntaxhighlight lang="python">from joblib import Parallel, delayed</syntaxhighlight>
We then can move our list-recursive example into our <code>joblib</code> structure. We wrap <code>Parallel(...)()</code> around the inside of the list-recursion and <code>delayed()</code> around the functions that will be repeated multiple times.

<syntaxhighlight lang="python">starttime = time.time()
r = Parallel(n_jobs=num_cores-2)(delayed(my_simulation)(k) for k in my_k)
print(str(time.time() - starttime) + " seconds")</syntaxhighlight>
<syntaxhighlight lang="python">0.5306367874145508 seconds</syntaxhighlight>
=== Add-ons ===

There are multiple add-ons that can be desirable

<ol style="list-style-type: decimal;">
<li>'''Verbosity''' can be added with a single parameter use (as seen below). This will provide more detailed output information.
<syntaxhighlight lang="python">r = Parallel(n_jobs=num_cores-2, verbose=10)(delayed(my_simulation)(k) for k in my_k)</syntaxhighlight></li>
<li>'''Multiple parameters''' can be passed with redefinitions of functions to take groupings. To replace multiple for-loops, often the function <code>itertools.product()</code> is used for the parameters.</li></ol>

== <code>ray</code> ==
The package <code>ray</code> is an open source project for parallel and distributed computing in Python. <code>ray</code> is often preferred over <code>multiprocess/multiprocessing</code> because of its efficient machine error handling methodology and its ability to handle larger data types. <code>ray</code> allows for aggregation of results/values, meaning the output of one task can be used as input for subsequent tasks seamlessly. The implementation for <code>ray</code> is not dissimilar from that of <code>multiprocess</code>.

=== Implementation ===
<syntaxhighlight lang="python">import ray
import os</syntaxhighlight>In order to parallelize computation with <code>ray</code>, you must first initialize an instance of <code>ray</code>. Just like with <code>multiprocess</code>, the instance relies on the number of CPUs available.<syntaxhighlight lang="python">
ray.init(num_cpus = os.cpu_count())
</syntaxhighlight>Additionally, in order for <code>ray</code> to work, the function <code>my_simulation</code> needs to have the decorator <code>@ray.remote</code> prior to its heading. <syntaxhighlight lang="python">
@ray.remote
def my_simulation(k):
"""the rest of the function code goes here"""
</syntaxhighlight>Once we have initialized an instance based on our number of available CPUs, and we have made sure our function has the proper decorator, we can parallelize computation as follows. The <code>.remote()</code> passes the variable into the function, even if the previous computation has not finished running, and the <code>.get()</code> compiles the results once finished.<syntaxhighlight lang="python">
starttime = time.time()
futures = [my_simulation.remote(k) for k in my_k]
results = ray.get(futures)
print(str(time.time() - starttime) + " seconds")
</syntaxhighlight>This <code>.remote()</code> piece is useful because you can pass previous computations into it in order to have a parallelized, autoregressive work flow.<syntaxhighlight>
0.13494515419006348 seconds
</syntaxhighlight>

= Jupyter Notebook: <code>ipyparallel</code>=

For those that heavily use Jupyter notebooks, the package <code>ipyparallel</code> works quite nicely. Some benefits of this package include its verbosity, its ability to serialize numpy arrays cleanly, and the fact that it can distribute across multiple machines. This package also works with <code>numba.jit</code> and <code>cython</code>.

Before opening your <code>jupyter notebook</code> you’ll want to make sure the extension is enabled - which can be done from the command line with the following command or inside the <code>jupyter notebook</code> extensions tab.

<syntaxhighlight lang="bash">ipcluster nbextension enable</syntaxhighlight>
You’ll then need to go to the “IPython Clusters” tab in <code>jupyter noteook</code> and start some parallel clusters.

==<code>ipyparallel</code> parallelization ==

<syntaxhighlight lang="python">import ipyparallel as ipp</syntaxhighlight>
For direct execution we define <code>dv</code> and provide it with what it needs.

<syntaxhighlight lang="python">rc = ipp.Client()
dv = rc[:] # direct view</syntaxhighlight>
=== Passing Things to Each Cluster ===

<code>ipyparallel</code> provides ways for the user to pass only the necessary items to the the clusters as to not overload them.

<ol style="list-style-type: decimal;">
<li>Passing packages all together
<syntaxhighlight lang="python">with dv.sync_imports():
from sklearn.linear_model import LinearRegression
# getting dependencies on the clusters</syntaxhighlight></li>
<li>Passing variables:
Below is just an example - we don’t need to pass <code>my_k</code> since <code>dv</code> will directly get it when we want to use it.
<syntaxhighlight lang="python">dv["my_k"] = my_k</syntaxhighlight></li>
<li>We can also use the <code>%px</code> magick to run lines of code on all servers.
<syntaxhighlight lang="python">%px import numpy as np</syntaxhighlight></li></ol>

=== Actual Functionality ===

The actual functionality is very similar to <code>multiprocess</code>/<code>multiprocessing</code> with <code>maps</code> and <code>apply</code>.

==== Synchronous ====

<syntaxhighlight lang="python">%%timeit
results = dv.map_sync(my_simulation, my_k)</syntaxhighlight>
<syntaxhighlight lang="python">201 ms ± 34.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)</syntaxhighlight>
==== Asynchronous ====
For asynchronous, you need to use <code>.get()</code> in order to get the results at a later time/when they come in, since the work is not done in order and is instead chunked for efficiency.<syntaxhighlight lang="python">%%timeit
results = dv.map_async(my_simulation, my_k).get() </syntaxhighlight>
<syntaxhighlight lang="python">209 ms ± 28.1 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)</syntaxhighlight>
= Just in Time Compilers + Parallelization =

Just-in-time compilation is the compiling of your computer code during execution instead of beforehand. This means bytecode is not translated to machine code until run time, and is then executed directly. Code is continuously analyzed to determine where compilation/recompilation would speed up performance. Used in conjunction with parallelization, this is an efficient way to improve computational speed.

== <code>numba</code> ==
<code>numba</code> is a high performance Python compiler that translates Python/numpy code into fast, efficient machine code. Python is traditionally regarded as a "slow" language; using <code>numba</code>, you can approach the speeds of languages such as C. <code>numba</code> is specifically designed to work well with numpy arrays.

=== Implementation ===
Similar to <code>ray</code>, <code>numba</code> requires a decorator <code>@njit (parallel = True)</code> before the function. We will define a different function to be parallelized here to demonstrate <code>numba</code> <code>prange</code> which allows for efficient parallelization across loops.<syntaxhighlight lang="python">
A = np.random.rand(3, 3)
B = np.random.rand(3, 3)

@njit(parallel = True)
def my_simulation(A, B):
C = np.empty_like(A)
for i in prange(A.shape[0]):
for j in range(A.shape[1]):
C[i, j] = A[i, j] + B[i, j]
return C
</syntaxhighlight><code>prange</code> should not be nested and should only ever be used in the outer loop. This is a very simple example, but for computationally complex looping problems, <code>numba</code> is highly efficient. However, you need to ensure that you are not creating or declaring a variable within a loop that cannot be accessed outside of the loop, or else parallelization will not work.

= Additional Topics and Resources =

# [https://ray.io/ <code>Ray</code> python package] allows paralleilization across multiple computers
# [https://people.duke.edu/~ccc14/sta-663-2018/notebooks/S14A_Parallel_Programming_Introduction.html Duke’s statistical computing course] provides a set of examples for multiple python parallelization packages |

Parallelization in python

2025-06-25T17:29:21Z

Ecucuzze: Added a section on ray

'''This article is still being developed - released earlier to align with Department seminar.'''

In Python, parallelization is a bit more of a “wild west” in terms of the number of parallelization packages available. This wiki will attempt to present minor examples of some common approaches, along with an outline of things to think about in selecting the library you end up using. In general, the process in Python involves:

# defining a function that captures what you want to have parallelized
# calling that function repeatedly in the the parallelization package way (which often looks like a list-recursive call).

There are a few different general frameworks to accomplish this.

# '''General wrappers''' define the cores you’ll be using and then wrap that around the function call in a list-recursive way.
# '''Just in time compilers''', in addition to providing parallelization tools, provide decorators (e.g. <code>@jit</code>) for functions that allow for just in time compilation of the Python function into lower level machine code (like <code>C++</code>).

We will also discuss Python packages that are built specifically to parallelize jobs inside '''Jupyter notebooks'''.

= Non-parallelized: Single Core Example =

We naturally want to compare a parallelization approach with the base single core analysis, which can be done with the code as follows:

<syntaxhighlight lang="python">#base for-loop
import numpy as np
from sklearn.linear_model import LinearRegression

my_k = list(np.arange(14)+1)*20

def my_simulation(k):
"""
estamine the polynomial fit of degree k's MSE for a linear
relationship

Parameters
----------
k : int
degree of polynomial fit

Returns
-------
mse of fit based on 30 points applied to a test set of 10000 points
"""
x = np.random.uniform(size=30,low=-3,high=3)
y = 2*x + 2* np.random.normal(size=30)
y = y.reshape(-1,1)
X_poly = np.concatenate([x.reshape(-1,1)**kk
for kk in np.arange(1,k+1,dtype = int)],
axis = 1)

x0 = np.random.uniform(size=10000,low=-3,high=3)
y0 = 2*x0 + 2*np.random.normal(size=10000)
y0 = y0.reshape(-1,1)
X0_poly = np.concatenate([x0.reshape(-1,1)**kk
for kk in np.arange(1,k+1,dtype = int)],
axis = 1)

my_lm = LinearRegression().fit(X_poly,y)
y0_hat = my_lm.predict(X0_poly)

return(np.mean((y0-y0_hat)**2))
</syntaxhighlight>
<syntaxhighlight lang="python">import time

starttime = time.time()
[my_simulation(k) for k in my_k]
print(str(time.time() - starttime) + " seconds")</syntaxhighlight>
<syntaxhighlight lang="python">0.9454693794250488 seconds</syntaxhighlight>
= General Python Parallelization Packages =

As previously mentioned, there are a lot of packages for general parallelization. Two commonly used ones are <code>joblib</code> and <code>multiprocessing</code>. <code>joblib</code> looks a bit more like list-recursion and <code>multiprocessing</code> uses functions like <code>.map()</code>.

== <code>multiprocessing</code>/<code>multiprocess</code> ==

The <code>multiprocessing</code> package is actually built into Python, and works in a pretty straightforward manner. It also allows for synchronous vs. asynchronous parallelization, where synchronous means the order of the jobs must be followed, one after the next, and asynchronous means multiple tasks can be executed in small parts in overlapping periods. Note that many statistical uses of parallelization do not need synchronous parallelization.

Since both approaches have the same syntax we’ll be using <code>multiprocess</code> (as it works with <code>ipython</code> and <code>jupyter notebooks</code>). <code>multiprocessing</code> can have problems with <code>ipython</code> and <code>jupyter notebook</code>s. In <code>jupyter notebook</code>s, if you want to use <code>multiprocessing</code> , it needs to be wrapped in <code>if name == "main"</code> to properly fork processes; you can also save your process is as a .py file and import it into your <code>jupyter notebook.</code>

=== Implementation ===

First, you need to identify the number of cores you have:

<syntaxhighlight lang="python">import multiprocess as mp
#Could also using import multiprocessing as mp; the syntax is the same

num_cores = mp.cpu_count()</syntaxhighlight>
==== Synchronous ====

<syntaxhighlight lang="python">starttime = time.time()
with mp.Pool(processes=num_cores-2) as p:
results = p.map(my_simulation, my_k)
print(str(time.time() - starttime) + " seconds")</syntaxhighlight>
<syntaxhighlight lang="python">0.7638578414916992 seconds</syntaxhighlight>
==== Asynchronous ====

<syntaxhighlight lang="python">starttime = time.time()
with mp.Pool(processes=num_cores-2) as p:
results = p.map_async(my_simulation, my_k).get()
print(str(time.time() - starttime) + " seconds")
</syntaxhighlight>
<syntaxhighlight lang="python">0.8183681964874268 seconds</syntaxhighlight>
== <code>joblib</code> ==

The <code>joblib</code> package is pretty straightforward and requires you to structure things in terms of list-recursion with a few wrappers to help the process work as desired. Just like with <code>multiprocessing</code>, the number of cores needs to be computed before proceeding.

=== Implementation ===

<syntaxhighlight lang="python">import joblib

num_cores = joblib.cpu_count()</syntaxhighlight>To use <code>joblib</code> functionality, we can import the functions <code>Parallel</code> and <code>delayed</code>. <code>Parallel</code> creates an instance for parallelization with a set number of jobs based on the core availability. <code>delayed</code> creates deferred function calls to distribute computations across multiple cores; this is the function that allows for simultaneous task execution.

<syntaxhighlight lang="python">from joblib import Parallel, delayed</syntaxhighlight>
We then can move our list-recursive example into our <code>joblib</code> structure. We wrap <code>Parallel(...)()</code> around the inside of the list-recursion and <code>delayed()</code> around the functions that will be repeated multiple times.

<syntaxhighlight lang="python">starttime = time.time()
r = Parallel(n_jobs=num_cores-2)(delayed(my_simulation)(k) for k in my_k)
print(str(time.time() - starttime) + " seconds")</syntaxhighlight>
<syntaxhighlight lang="python">0.5306367874145508 seconds</syntaxhighlight>
=== Add-ons ===

There are multiple add-ons that can be desirable

<ol style="list-style-type: decimal;">
<li>'''Verbosity''' can be added with a single parameter use (as seen below). This will provide more detailed output information.
<syntaxhighlight lang="python">r = Parallel(n_jobs=num_cores-2, verbose=10)(delayed(my_simulation)(k) for k in my_k)</syntaxhighlight></li>
<li>'''Multiple parameters''' can be passed with redefinitions of functions to take groupings. To replace multiple for-loops, often the function <code>itertools.product()</code> is used for the parameters.</li></ol>

== <code>ray</code> ==
The package <code>ray</code> is an open source project for parallel and distributed computing in Python. <code>ray</code> is often preferred over <code>multiprocess/multiprocessing</code> because of its efficient machine error handling methodology and its ability to handle larger data types. <code>ray</code> allows for aggregation of results/values, meaning the output of one task can be used as input for subsequent tasks seamlessly. The implementation for <code>ray</code> is not dissimilar from that of <code>multiprocess</code>.

=== Implementation ===
<syntaxhighlight lang="python">import ray
import os</syntaxhighlight>In order to parallelize computation with <code>ray</code>, you must first initialize an instance of <code>ray</code>. Just like with <code>multiprocess</code>, the instance relies on the number of CPUs available.<syntaxhighlight lang="python">
ray.init(num_cpus = os.cpu_count())
</syntaxhighlight>Additionally, in order for <code>ray</code> to work, the function <code>my_simulation</code> needs to have the docker <code>@ray.remote</code> prior to its heading. <syntaxhighlight lang="python">
@ray.remote
def my_simulation(k):
"""the rest of the function code goes here"""
</syntaxhighlight>Once we have initialized an instance based on our number of available CPUs, and we have made sure our function has the proper docker, we can parallelize computation as follows. The <code>.remote()</code> passes the variable into the function, even if the previous computation has not finished running, and the <code>.get()</code> compiles the results once finished.<syntaxhighlight lang="python">
starttime = time.time()
futures = [my_simulation.remote(k) for k in my_k]
results = ray.get(futures)
print(str(time.time() - starttime) + " seconds")
</syntaxhighlight>This <code>.remote()</code> piece is useful because you can pass previous computations into it in order to have a parallelized, autoregressive work flow.<syntaxhighlight>
0.13494515419006348 seconds
</syntaxhighlight>

= Jupyter Notebook: <code>ipyparallel</code>=

For those that heavily use Jupyter notebooks, the package <code>ipyparallel</code> works quite nicely. Some benefits of this package include its verbosity, its ability to serialize numpy arrays cleanly, and the fact that it can distribute across multiple machines. This package also works with <code>numba.jit</code> and <code>cython</code>.

Before opening your <code>jupyter notebook</code> you’ll want to make sure the extension is enabled - which can be done from the command line with the following command or inside the <code>jupyter notebook</code> extensions tab.

<syntaxhighlight lang="bash">ipcluster nbextension enable</syntaxhighlight>
You’ll then need to go to the “IPython Clusters” tab in <code>jupyter noteook</code> and start some parallel clusters.

==<code>ipyparallel</code> parallelization ==

<syntaxhighlight lang="python">import ipyparallel as ipp</syntaxhighlight>
For direct execution we define <code>dv</code> and provide it with what it needs.

<syntaxhighlight lang="python">rc = ipp.Client()
dv = rc[:] # direct view</syntaxhighlight>
=== Passing Things to Each Cluster ===

<code>ipyparallel</code> provides ways for the user to pass only the necessary items to the the clusters as to not overload them.

<ol style="list-style-type: decimal;">
<li>Passing packages all together
<syntaxhighlight lang="python">with dv.sync_imports():
from sklearn.linear_model import LinearRegression
# getting dependencies on the clusters</syntaxhighlight></li>
<li>Passing variables:
Below is just an example - we don’t need to pass <code>my_k</code> since <code>dv</code> will directly get it when we want to use it.
<syntaxhighlight lang="python">dv["my_k"] = my_k</syntaxhighlight></li>
<li>We can also use the <code>%px</code> magick to run lines of code on all servers.
<syntaxhighlight lang="python">%px import numpy as np</syntaxhighlight></li></ol>

=== Actual Functionality ===

The actual functionality is very similar to <code>multiprocess</code>/<code>multiprocessing</code> with <code>maps</code> and <code>apply</code>.

==== Synchronous ====

<syntaxhighlight lang="python">%%timeit
results = dv.map_sync(my_simulation, my_k)</syntaxhighlight>
<syntaxhighlight lang="python">201 ms ± 34.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)</syntaxhighlight>
==== Asynchronous ====
For asynchronous, you need to use <code>.get()</code> in order to get the results at a later time/when they come in, since the work is not done in order and is instead chunked for efficiency.<syntaxhighlight lang="python">%%timeit
results = dv.map_async(my_simulation, my_k).get() </syntaxhighlight>
<syntaxhighlight lang="python">209 ms ± 28.1 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)</syntaxhighlight>
= Just in Time compilers + parallelization =

TODO: <code>numba</code> and <code>cython</code>. The idea of these is to compile the function you will be using the first time you use it and then never compile it again (speeding up the actual use of the function). That is then combined with standard parallelization
= Additional Topics and Resources =

# [https://ray.io/ <code>Ray</code> python package] allows paralleilization across multiple computers
# [https://people.duke.edu/~ccc14/sta-663-2018/notebooks/S14A_Parallel_Programming_Introduction.html Duke’s statistical computing course] provides a set of examples for multiple python parallelization packages |

Parallelization in python

2025-06-25T16:31:26Z

Ecucuzze: Updated the joblibs times to match what I am currently getting

'''This article is still being developed - released earlier to align with Department seminar.'''

In Python, parallelization is a bit more of a “wild west” in terms of the number of parallelization packages available. This wiki will attempt to present minor examples of some common approaches, along with an outline of things to think about in selecting the library you end up using. In general, the process in Python involves:

# defining a function that captures what you want to have parallelized
# calling that function repeatedly in the the parallelization package way (which often looks like a list-recursive call).

There are a few different general frameworks to accomplish this.

# '''General wrappers''' define the cores you’ll be using and then wrap that around the function call in a list-recursive way.
# '''Just in time compilers''', in addition to providing parallelization tools, provide decorators (e.g. <code>@jit</code>) for functions that allow for just in time compilation of the Python function into lower level machine code (like <code>C++</code>).

We will also discuss Python packages that are built specifically to parallelize jobs inside '''Jupyter notebooks'''.

= Non-parallelized: Single Core Example =

We naturally want to compare a parallelization approach with the base single core analysis, which can be done with the code as follows:

<syntaxhighlight lang="python">#base for-loop
import numpy as np
from sklearn.linear_model import LinearRegression

my_k = list(np.arange(14)+1)*20

def my_simulation(k):
"""
estamine the polynomial fit of degree k's MSE for a linear
relationship

Parameters
----------
k : int
degree of polynomial fit

Returns
-------
mse of fit based on 30 points applied to a test set of 10000 points
"""
x = np.random.uniform(size=30,low=-3,high=3)
y = 2*x + 2* np.random.normal(size=30)
y = y.reshape(-1,1)
X_poly = np.concatenate([x.reshape(-1,1)**kk
for kk in np.arange(1,k+1,dtype = int)],
axis = 1)

x0 = np.random.uniform(size=10000,low=-3,high=3)
y0 = 2*x0 + 2*np.random.normal(size=10000)
y0 = y0.reshape(-1,1)
X0_poly = np.concatenate([x0.reshape(-1,1)**kk
for kk in np.arange(1,k+1,dtype = int)],
axis = 1)

my_lm = LinearRegression().fit(X_poly,y)
y0_hat = my_lm.predict(X0_poly)

return(np.mean((y0-y0_hat)**2))
</syntaxhighlight>
<syntaxhighlight lang="python">import time

starttime = time.time()
[my_simulation(k) for k in my_k]
print(str(time.time() - starttime) + " seconds")</syntaxhighlight>
<syntaxhighlight lang="python">0.9454693794250488 seconds</syntaxhighlight>
= General Python Parallelization Packages =

As previously mentioned, there are a lot of packages for general parallelization. Two commonly used ones are <code>joblib</code> and <code>multiprocessing</code>. <code>joblib</code> looks a bit more like list-recursion and <code>multiprocessing</code> uses functions like <code>.map()</code>.

== <code>multiprocessing</code>/<code>multiprocess</code> ==

The <code>multiprocessing</code> package is actually built into Python, and works in a pretty straightforward manner. It also allows for synchronous vs. asynchronous parallelization, where synchronous means the order of the jobs must be followed, one after the next, and asynchronous means multiple tasks can be executed in small parts in overlapping periods. Note that many statistical uses of parallelization do not need synchronous parallelization.

Since both approaches have the same syntax we’ll be using <code>multiprocess</code> (as it works with <code>ipython</code> and <code>jupyter notebooks</code>). <code>multiprocessing</code> can have problems with <code>ipython</code> and <code>jupyter notebook</code>s. In <code>jupyter notebook</code>s, if you want to use <code>multiprocessing</code> , it needs to be wrapped in <code>if name == "main"</code> to properly fork processes; you can also save your process is as a .py file and import it into your <code>jupyter notebook.</code>

=== Implementation ===

First, you need to identify the number of cores you have:

<syntaxhighlight lang="python">import multiprocess as mp
#Could also using import multiprocessing as mp; the syntax is the same

num_cores = mp.cpu_count()</syntaxhighlight>
==== Synchronous ====

<syntaxhighlight lang="python">starttime = time.time()
with mp.Pool(processes=num_cores-2) as p:
results = p.map(my_simulation, my_k)
print(str(time.time() - starttime) + " seconds")</syntaxhighlight>
<syntaxhighlight lang="python">0.7638578414916992 seconds</syntaxhighlight>
==== Asynchronous ====

<syntaxhighlight lang="python">starttime = time.time()
with mp.Pool(processes=num_cores-2) as p:
results = p.map_async(my_simulation, my_k).get()
print(str(time.time() - starttime) + " seconds")
</syntaxhighlight>
<syntaxhighlight lang="python">0.8183681964874268 seconds</syntaxhighlight>
== <code>joblib</code> ==

The <code>joblib</code> package is pretty straightforward and requires you to structure things in terms of list-recursion with a few wrappers to help the process work as desired. Just like with <code>multiprocessing</code>, the number of cores needs to be computed before proceeding.

=== Implementation ===

<syntaxhighlight lang="python">import joblib

num_cores = joblib.cpu_count()</syntaxhighlight>To use <code>joblib</code> functionality, we can import the functions <code>Parallel</code> and <code>delayed</code>. <code>Parallel</code> creates an instance for parallelization with a set number of jobs based on the core availability. <code>delayed</code> creates deferred function calls to distribute computations across multiple cores; this is the function that allows for simultaneous task execution.

<syntaxhighlight lang="python">from joblib import Parallel, delayed</syntaxhighlight>
We then can move our list-recursive example into our <code>joblib</code> structure. We wrap <code>Parallel(...)()</code> around the inside of the list-recursion and <code>delayed()</code> around the functions that will be repeated multiple times.

<syntaxhighlight lang="python">starttime = time.time()
r = Parallel(n_jobs=num_cores-2)(delayed(my_simulation)(k) for k in my_k)
print(str(time.time() - starttime) + " seconds")</syntaxhighlight>
<syntaxhighlight lang="python">0.5306367874145508 seconds</syntaxhighlight>
=== Add-ons ===

There are multiple add-ons that can be desirable

<ol style="list-style-type: decimal;">
<li>'''Verbosity''' can be added with a single parameter use (as seen below). This will provide more detailed output information.
<syntaxhighlight lang="python">r = Parallel(n_jobs=num_cores-2, verbose=10)(delayed(my_simulation)(k) for k in my_k)</syntaxhighlight></li>
<li>'''Multiple parameters''' can be passed with redefinitions of functions to take groupings. To replace multiple for-loops, often the function <code>itertools.product()</code> is used for the parameters.</li></ol>

= Jupyter Notebook: <code>ipyparallel</code>=

For those that heavily use Jupyter notebooks, the package <code>ipyparallel</code> works quite nicely. Some benefits of this package include its verbosity, its ability to serialize numpy arrays cleanly, and the fact that it can distribute across multiple machines. This package also works with <code>numba.jit</code> and <code>cython</code>.

Before opening your <code>jupyter notebook</code> you’ll want to make sure the extension is enabled - which can be done from the command line with the following command or inside the <code>jupyter notebook</code> extensions tab.

<syntaxhighlight lang="bash">ipcluster nbextension enable</syntaxhighlight>
You’ll then need to go to the “IPython Clusters” tab in <code>jupyter noteook</code> and start some parallel clusters.

==<code>ipyparallel</code> parallelization ==

<syntaxhighlight lang="python">import ipyparallel as ipp</syntaxhighlight>
For direct execution we define <code>dv</code> and provide it with what it needs.

<syntaxhighlight lang="python">rc = ipp.Client()
dv = rc[:] # direct view</syntaxhighlight>
=== Passing Things to Each Cluster ===

<code>ipyparallel</code> provides ways for the user to pass only the necessary items to the the clusters as to not overload them.

<ol style="list-style-type: decimal;">
<li>Passing packages all together
<syntaxhighlight lang="python">with dv.sync_imports():
from sklearn.linear_model import LinearRegression
# getting dependencies on the clusters</syntaxhighlight></li>
<li>Passing variables:
Below is just an example - we don’t need to pass <code>my_k</code> since <code>dv</code> will directly get it when we want to use it.
<syntaxhighlight lang="python">dv["my_k"] = my_k</syntaxhighlight></li>
<li>We can also use the <code>%px</code> magick to run lines of code on all servers.
<syntaxhighlight lang="python">%px import numpy as np</syntaxhighlight></li></ol>

=== Actual Functionality ===

The actual functionality is very similar to <code>multiprocess</code>/<code>multiprocessing</code> with <code>maps</code> and <code>apply</code>.

==== Synchronous ====

<syntaxhighlight lang="python">%%timeit
results = dv.map_sync(my_simulation, my_k)</syntaxhighlight>
<syntaxhighlight lang="python">201 ms ± 34.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)</syntaxhighlight>
==== Asynchronous ====
For asynchronous, you need to use <code>.get()</code> in order to get the results at a later time/when they come in, since the work is not done in order and is instead chunked for efficiency.<syntaxhighlight lang="python">%%timeit
results = dv.map_async(my_simulation, my_k).get() </syntaxhighlight>
<syntaxhighlight lang="python">209 ms ± 28.1 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)</syntaxhighlight>
= Just in Time compilers + parallelization =

TODO: <code>numba</code> and <code>cython</code>. The idea of these is to compile the function you will be using the first time you use it and then never compile it again (speeding up the actual use of the function). That is then combined with standard parallelization
= Additional Topics and Resources =

# [https://ray.io/ <code>Ray</code> python package] allows paralleilization across multiple computers
# [https://people.duke.edu/~ccc14/sta-663-2018/notebooks/S14A_Parallel_Programming_Introduction.html Duke’s statistical computing course] provides a set of examples for multiple python parallelization packages |

Parallelization in python

2025-06-25T16:24:59Z

Ecucuzze: Updated with the new times I'm getting so the results are consistent

'''This article is still being developed - released earlier to align with Department seminar.'''

In Python, parallelization is a bit more of a “wild west” in terms of the number of parallelization packages available. This wiki will attempt to present minor examples of some common approaches, along with an outline of things to think about in selecting the library you end up using. In general, the process in Python involves:

# defining a function that captures what you want to have parallelized
# calling that function repeatedly in the the parallelization package way (which often looks like a list-recursive call).

There are a few different general frameworks to accomplish this.

# '''General wrappers''' define the cores you’ll be using and then wrap that around the function call in a list-recursive way.
# '''Just in time compilers''', in addition to providing parallelization tools, provide decorators (e.g. <code>@jit</code>) for functions that allow for just in time compilation of the Python function into lower level machine code (like <code>C++</code>).

We will also discuss Python packages that are built specifically to parallelize jobs inside '''Jupyter notebooks'''.

= Non-parallelized: Single Core Example =

We naturally want to compare a parallelization approach with the base single core analysis, which can be done with the code as follows:

<syntaxhighlight lang="python">#base for-loop
import numpy as np
from sklearn.linear_model import LinearRegression

my_k = list(np.arange(14)+1)*20

def my_simulation(k):
"""
estamine the polynomial fit of degree k's MSE for a linear
relationship

Parameters
----------
k : int
degree of polynomial fit

Returns
-------
mse of fit based on 30 points applied to a test set of 10000 points
"""
x = np.random.uniform(size=30,low=-3,high=3)
y = 2*x + 2* np.random.normal(size=30)
y = y.reshape(-1,1)
X_poly = np.concatenate([x.reshape(-1,1)**kk
for kk in np.arange(1,k+1,dtype = int)],
axis = 1)

x0 = np.random.uniform(size=10000,low=-3,high=3)
y0 = 2*x0 + 2*np.random.normal(size=10000)
y0 = y0.reshape(-1,1)
X0_poly = np.concatenate([x0.reshape(-1,1)**kk
for kk in np.arange(1,k+1,dtype = int)],
axis = 1)

my_lm = LinearRegression().fit(X_poly,y)
y0_hat = my_lm.predict(X0_poly)

return(np.mean((y0-y0_hat)**2))
</syntaxhighlight>
<syntaxhighlight lang="python">import time

starttime = time.time()
[my_simulation(k) for k in my_k]
print(str(time.time() - starttime) + " seconds")</syntaxhighlight>
<syntaxhighlight lang="python">0.9454693794250488 seconds</syntaxhighlight>
= General Python Parallelization Packages =

As previously mentioned, there are a lot of packages for general parallelization. Two commonly used ones are <code>joblib</code> and <code>multiprocessing</code>. <code>joblib</code> looks a bit more like list-recursion and <code>multiprocessing</code> uses functions like <code>.map()</code>.

== <code>multiprocessing</code>/<code>multiprocess</code> ==

The <code>multiprocessing</code> package is actually built into Python, and works in a pretty straightforward manner. It also allows for synchronous vs. asynchronous parallelization, where synchronous means the order of the jobs must be followed, one after the next, and asynchronous means multiple tasks can be executed in small parts in overlapping periods. Note that many statistical uses of parallelization do not need synchronous parallelization.

Since both approaches have the same syntax we’ll be using <code>multiprocess</code> (as it works with <code>ipython</code> and <code>jupyter notebooks</code>). <code>multiprocessing</code> can have problems with <code>ipython</code> and <code>jupyter notebook</code>s. In <code>jupyter notebook</code>s, if you want to use <code>multiprocessing</code> , it needs to be wrapped in <code>if name == "main"</code> to properly fork processes; you can also save your process is as a .py file and import it into your <code>jupyter notebook.</code>

=== Implementation ===

First, you need to identify the number of cores you have:

<syntaxhighlight lang="python">import multiprocess as mp
#Could also using import multiprocessing as mp; the syntax is the same

num_cores = mp.cpu_count()</syntaxhighlight>
==== Synchronous ====

<syntaxhighlight lang="python">%%timeit

with mp.Pool(processes=num_cores-2) as p:
results = p.map(my_simulation, my_k)</syntaxhighlight>
<syntaxhighlight lang="python">209 ms ± 24.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)</syntaxhighlight>
==== Asynchronous ====

<syntaxhighlight lang="python">%%timeit

with mp.Pool(processes=num_cores-2) as p:
results = p.map_async(my_simulation, my_k).get() # not clear why I needed to use .get() to get something returnable for async</syntaxhighlight>
<syntaxhighlight lang="python">48.9 ms ± 3.08 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)</syntaxhighlight>
== <code>joblib</code> ==

The <code>joblib</code> package is pretty straightforward and requires you to structure things in terms of list-recursion with a few wrappers to help the process work as desired. Just like with <code>multiprocessing</code>, the number of cores needs to be computed before proceeding.

=== Implementation ===

<syntaxhighlight lang="python">import joblib

num_cores = joblib.cpu_count()</syntaxhighlight>To use <code>joblib</code> functionality, we can import the functions <code>Parallel</code> and <code>delayed</code>. <code>Parallel</code> creates an instance for parallelization with a set number of jobs based on the core availability. <code>delayed</code> creates deferred function calls to distribute computations across multiple cores; this is the function that allows for simultaneous task execution.

<syntaxhighlight lang="python">from joblib import Parallel, delayed</syntaxhighlight>
We then can move our list-recursive example into our <code>joblib</code> structure. We wrap <code>Parallel(...)()</code> around the inside of the list-recursion and <code>delayed()</code> around the functions that will be repeated multiple times.

<syntaxhighlight lang="python">%% timeit

r = Parallel(n_jobs=num_cores-2)(delayed(my_simulation)(k) for k in my_k)</syntaxhighlight>
<syntaxhighlight lang="python">172 ms ± 23.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)</syntaxhighlight>
=== Add-ons ===

There are multiple add-ons that can be desirable

<ol style="list-style-type: decimal;">
<li>'''Verbosity''' can be added with a single parameter use (as seen below). This will provide more detailed output information.
<syntaxhighlight lang="python">r = Parallel(n_jobs=num_cores-2, verbose=10)(delayed(my_simulation)(k) for k in my_k)</syntaxhighlight></li>
<li>'''Multiple parameters''' can be passed with redefinitions of functions to take groupings. To replace multiple for-loops, often the function <code>itertools.product()</code> is used for the parameters.</li></ol>

= Jupyter Notebook: <code>ipyparallel</code>=

For those that heavily use Jupyter notebooks, the package <code>ipyparallel</code> works quite nicely. Some benefits of this package include its verbosity, its ability to serialize numpy arrays cleanly, and the fact that it can distribute across multiple machines. This package also works with <code>numba.jit</code> and <code>cython</code>.

Before opening your <code>jupyter notebook</code> you’ll want to make sure the extension is enabled - which can be done from the command line with the following command or inside the <code>jupyter notebook</code> extensions tab.

<syntaxhighlight lang="bash">ipcluster nbextension enable</syntaxhighlight>
You’ll then need to go to the “IPython Clusters” tab in <code>jupyter noteook</code> and start some parallel clusters.

==<code>ipyparallel</code> parallelization ==

<syntaxhighlight lang="python">import ipyparallel as ipp</syntaxhighlight>
For direct execution we define <code>dv</code> and provide it with what it needs.

<syntaxhighlight lang="python">rc = ipp.Client()
dv = rc[:] # direct view</syntaxhighlight>
=== Passing Things to Each Cluster ===

<code>ipyparallel</code> provides ways for the user to pass only the necessary items to the the clusters as to not overload them.

<ol style="list-style-type: decimal;">
<li>Passing packages all together
<syntaxhighlight lang="python">with dv.sync_imports():
from sklearn.linear_model import LinearRegression
# getting dependencies on the clusters</syntaxhighlight></li>
<li>Passing variables:
Below is just an example - we don’t need to pass <code>my_k</code> since <code>dv</code> will directly get it when we want to use it.
<syntaxhighlight lang="python">dv["my_k"] = my_k</syntaxhighlight></li>
<li>We can also use the <code>%px</code> magick to run lines of code on all servers.
<syntaxhighlight lang="python">%px import numpy as np</syntaxhighlight></li></ol>

=== Actual Functionality ===

The actual functionality is very similar to <code>multiprocess</code>/<code>multiprocessing</code> with <code>maps</code> and <code>apply</code>.

==== Synchronous ====

<syntaxhighlight lang="python">%%timeit
results = dv.map_sync(my_simulation, my_k)</syntaxhighlight>
<syntaxhighlight lang="python">201 ms ± 34.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)</syntaxhighlight>
==== Asynchronous ====
For asynchronous, you need to use <code>.get()</code> in order to get the results at a later time/when they come in, since the work is not done in order and is instead chunked for efficiency.<syntaxhighlight lang="python">%%timeit
results = dv.map_async(my_simulation, my_k).get() </syntaxhighlight>
<syntaxhighlight lang="python">209 ms ± 28.1 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)</syntaxhighlight>
= Just in Time compilers + parallelization =

TODO: <code>numba</code> and <code>cython</code>. The idea of these is to compile the function you will be using the first time you use it and then never compile it again (speeding up the actual use of the function). That is then combined with standard parallelization
= Additional Topics and Resources =

# [https://ray.io/ <code>Ray</code> python package] allows paralleilization across multiple computers
# [https://people.duke.edu/~ccc14/sta-663-2018/notebooks/S14A_Parallel_Programming_Introduction.html Duke’s statistical computing course] provides a set of examples for multiple python parallelization packages |

Parallelization in python

2025-06-25T16:00:00Z

Ecucuzze:

'''This article is still being developed - released earlier to align with Department seminar.'''

In Python, parallelization is a bit more of a “wild west” in terms of the number of parallelization packages available. This wiki will attempt to present minor examples of some common approaches, along with an outline of things to think about in selecting the library you end up using. In general, the process in Python involves:

# defining a function that captures what you want to have parallelized
# calling that function repeatedly in the the parallelization package way (which often looks like a list-recursive call).

There are a few different general frameworks to accomplish this.

# '''General wrappers''' define the cores you’ll be using and then wrap that around the function call in a list-recursive way.
# '''Just in time compilers''', in addition to providing parallelization tools, provide decorators (e.g. <code>@jit</code>) for functions that allow for just in time compilation of the Python function into lower level machine code (like <code>C++</code>).

We will also discuss Python packages that are built specifically to parallelize jobs inside '''Jupyter notebooks'''.

= Non-parallelized: Single Core Example =

We naturally want to compare a parallelization approach with the base single core analysis, which can be done with the code as follows:

<syntaxhighlight lang="python">#base for-loop
import numpy as np
from sklearn.linear_model import LinearRegression

my_k = list(np.arange(14)+1)*20

def my_simulation(k):
"""
estamine the polynomial fit of degree k's MSE for a linear
relationship

Parameters
----------
k : int
degree of polynomial fit

Returns
-------
mse of fit based on 30 points applied to a test set of 10000 points
"""
x = np.random.uniform(size=30,low=-3,high=3)
y = 2*x + 2* np.random.normal(size=30)
y = y.reshape(-1,1)
X_poly = np.concatenate([x.reshape(-1,1)**kk
for kk in np.arange(1,k+1,dtype = int)],
axis = 1)

x0 = np.random.uniform(size=10000,low=-3,high=3)
y0 = 2*x0 + 2*np.random.normal(size=10000)
y0 = y0.reshape(-1,1)
X0_poly = np.concatenate([x0.reshape(-1,1)**kk
for kk in np.arange(1,k+1,dtype = int)],
axis = 1)

my_lm = LinearRegression().fit(X_poly,y)
y0_hat = my_lm.predict(X0_poly)

return(np.mean((y0-y0_hat)**2))
</syntaxhighlight>
<syntaxhighlight lang="python">%%timeit -n3

[my_simulation(k) for k in my_k]</syntaxhighlight>
<syntaxhighlight lang="python">676 ms ± 37.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)</syntaxhighlight>
= General Python Parallelization Packages =

As previously mentioned, there are a lot of packages for general parallelization. Two commonly used ones are <code>joblib</code> and <code>multiprocessing</code>. <code>joblib</code> looks a bit more like list-recursion and <code>multiprocessing</code> uses functions like <code>.map()</code>.

== <code>multiprocessing</code>/<code>multiprocess</code> ==

The <code>multiprocessing</code> package is actually built into Python, and works in a pretty straightforward manner. It also allows for synchronous vs. asynchronous parallelization, where synchronous means the order of the jobs must be followed, one after the next, and asynchronous means multiple tasks can be executed in small parts in overlapping periods. Note that many statistical uses of parallelization do not need synchronous parallelization.

Since both approaches have the same syntax we’ll be using <code>multiprocess</code> (as it works with <code>ipython</code> and <code>jupyter notebooks</code>). <code>multiprocessing</code> can have problems with <code>ipython</code> and <code>jupyter notebook</code>s. In <code>jupyter notebook</code>s, if you want to use <code>multiprocessing</code> , it needs to be wrapped in <code>if name == "main"</code> to properly fork processes; you can also save your process is as a .py file and import it into your <code>jupyter notebook.</code>

=== Implementation ===

First, you need to identify the number of cores you have:

<syntaxhighlight lang="python">import multiprocess as mp
#Could also using import multiprocessing as mp; the syntax is the same

num_cores = mp.cpu_count()</syntaxhighlight>
==== Synchronous ====

<syntaxhighlight lang="python">%%timeit

with mp.Pool(processes=num_cores-2) as p:
results = p.map(my_simulation, my_k)</syntaxhighlight>
<syntaxhighlight lang="python">209 ms ± 24.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)</syntaxhighlight>
==== Asynchronous ====

<syntaxhighlight lang="python">%%timeit

with mp.Pool(processes=num_cores-2) as p:
results = p.map_async(my_simulation, my_k).get() # not clear why I needed to use .get() to get something returnable for async</syntaxhighlight>
<syntaxhighlight lang="python">48.9 ms ± 3.08 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)</syntaxhighlight>
== <code>joblib</code> ==

The <code>joblib</code> package is pretty straightforward and requires you to structure things in terms of list-recursion with a few wrappers to help the process work as desired. Just like with <code>multiprocessing</code>, the number of cores needs to be computed before proceeding.

=== Implementation ===

<syntaxhighlight lang="python">import joblib

num_cores = joblib.cpu_count()</syntaxhighlight>To use <code>joblib</code> functionality, we can import the functions <code>Parallel</code> and <code>delayed</code>. <code>Parallel</code> creates an instance for parallelization with a set number of jobs based on the core availability. <code>delayed</code> creates deferred function calls to distribute computations across multiple cores; this is the function that allows for simultaneous task execution.

<syntaxhighlight lang="python">from joblib import Parallel, delayed</syntaxhighlight>
We then can move our list-recursive example into our <code>joblib</code> structure. We wrap <code>Parallel(...)()</code> around the inside of the list-recursion and <code>delayed()</code> around the functions that will be repeated multiple times.

<syntaxhighlight lang="python">%% timeit

r = Parallel(n_jobs=num_cores-2)(delayed(my_simulation)(k) for k in my_k)</syntaxhighlight>
<syntaxhighlight lang="python">172 ms ± 23.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)</syntaxhighlight>
=== Add-ons ===

There are multiple add-ons that can be desirable

<ol style="list-style-type: decimal;">
<li>'''Verbosity''' can be added with a single parameter use (as seen below). This will provide more detailed output information.
<syntaxhighlight lang="python">r = Parallel(n_jobs=num_cores-2, verbose=10)(delayed(my_simulation)(k) for k in my_k)</syntaxhighlight></li>
<li>'''Multiple parameters''' can be passed with redefinitions of functions to take groupings. To replace multiple for-loops, often the function <code>itertools.product()</code> is used for the parameters.</li></ol>

= Jupyter Notebook: <code>ipyparallel</code>=

For those that heavily use Jupyter notebooks, the package <code>ipyparallel</code> works quite nicely. Some benefits of this package include its verbosity, its ability to serialize numpy arrays cleanly, and the fact that it can distribute across multiple machines. This package also works with <code>numba.jit</code> and <code>cython</code>.

Before opening your <code>jupyter notebook</code> you’ll want to make sure the extension is enabled - which can be done from the command line with the following command or inside the <code>jupyter notebook</code> extensions tab.

<syntaxhighlight lang="bash">ipcluster nbextension enable</syntaxhighlight>
You’ll then need to go to the “IPython Clusters” tab in <code>jupyter noteook</code> and start some parallel clusters.

==<code>ipyparallel</code> parallelization ==

<syntaxhighlight lang="python">import ipyparallel as ipp</syntaxhighlight>
For direct execution we define <code>dv</code> and provide it with what it needs.

<syntaxhighlight lang="python">rc = ipp.Client()
dv = rc[:] # direct view</syntaxhighlight>
=== Passing Things to Each Cluster ===

<code>ipyparallel</code> provides ways for the user to pass only the necessary items to the the clusters as to not overload them.

<ol style="list-style-type: decimal;">
<li>Passing packages all together
<syntaxhighlight lang="python">with dv.sync_imports():
from sklearn.linear_model import LinearRegression
# getting dependencies on the clusters</syntaxhighlight></li>
<li>Passing variables:
Below is just an example - we don’t need to pass <code>my_k</code> since <code>dv</code> will directly get it when we want to use it.
<syntaxhighlight lang="python">dv["my_k"] = my_k</syntaxhighlight></li>
<li>We can also use the <code>%px</code> magick to run lines of code on all servers.
<syntaxhighlight lang="python">%px import numpy as np</syntaxhighlight></li></ol>

=== Actual Functionality ===

The actual functionality is very similar to <code>multiprocess</code>/<code>multiprocessing</code> with <code>maps</code> and <code>apply</code>.

==== Synchronous ====

<syntaxhighlight lang="python">%%timeit
results = dv.map_sync(my_simulation, my_k)</syntaxhighlight>
<syntaxhighlight lang="python">201 ms ± 34.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)</syntaxhighlight>
==== Asynchronous ====
For asynchronous, you need to use <code>.get()</code> in order to get the results at a later time/when they come in, since the work is not done in order and is instead chunked for efficiency.<syntaxhighlight lang="python">%%timeit
results = dv.map_async(my_simulation, my_k).get() </syntaxhighlight>
<syntaxhighlight lang="python">209 ms ± 28.1 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)</syntaxhighlight>
= Just in Time compilers + parallelization =

TODO: <code>numba</code> and <code>cython</code>. The idea of these is to compile the function you will be using the first time you use it and then never compile it again (speeding up the actual use of the function). That is then combined with standard parallelization
= Additional Topics and Resources =

# [https://ray.io/ <code>Ray</code> python package] allows paralleilization across multiple computers
# [https://people.duke.edu/~ccc14/sta-663-2018/notebooks/S14A_Parallel_Programming_Introduction.html Duke’s statistical computing course] provides a set of examples for multiple python parallelization packages |

Parallelization in python

2025-06-25T14:49:05Z

Ecucuzze: Cleaned up and added further explanation to multiprocessing and joblib

'''This article is still being developed - released earlier to align with Department seminar.'''

In Python, parallelization is a bit more of a “wild west” in terms of the number of parallelization packages available. This wiki will attempt to present minor examples of some common approaches, along with an outline of things to think about in selecting the library you end up using. In general, the process in Python involves:

# defining a function that captures what you want to have parallelized
# calling that function repeatedly in the the parallelization package way (which often looks like a list-recursive call).

There are a few different general frameworks to accomplish this.

# '''General wrappers''' define the cores you’ll be using and then wrap that around the function call in a list-recursive way.
# '''Just in time compilers''', in addition to providing parallelization tools, provide decorators (e.g. <code>@jit</code>) for functions that allow for just in time compilation of the Python function into lower level machine code (like <code>C++</code>).

We will also discuss Python packages that are built specifically to parallelize jobs inside '''Jupyter notebooks'''.

= Non-parallelized: Single Core Example =

We naturally want to compare a parallelization approach with the base single core analysis, which can be done with the code as follows:

<syntaxhighlight lang="python">#base for-loop
import numpy as np
from sklearn.linear_model import LinearRegression

my_k = list(np.arange(14)+1)*20

def my_simulation(k):
"""
estamine the polynomial fit of degree k's MSE for a linear
relationship

Parameters
----------
k : int
degree of polynomial fit

Returns
-------
mse of fit based on 30 points applied to a test set of 10000 points
"""
x = np.random.uniform(size=30,low=-3,high=3)
y = 2*x + 2* np.random.normal(size=30)
y = y.reshape(-1,1)
X_poly = np.concatenate([x.reshape(-1,1)**kk
for kk in np.arange(1,k+1,dtype = int)],
axis = 1)

x0 = np.random.uniform(size=10000,low=-3,high=3)
y0 = 2*x0 + 2*np.random.normal(size=10000)
y0 = y0.reshape(-1,1)
X0_poly = np.concatenate([x0.reshape(-1,1)**kk
for kk in np.arange(1,k+1,dtype = int)],
axis = 1)

my_lm = LinearRegression().fit(X_poly,y)
y0_hat = my_lm.predict(X0_poly)

return(np.mean((y0-y0_hat)**2))
</syntaxhighlight>
<syntaxhighlight lang="python">%%timeit -n3

[my_simulation(k) for k in my_k]</syntaxhighlight>
<syntaxhighlight lang="python">676 ms ± 37.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)</syntaxhighlight>
= General Python Parallelization Packages =

As previously mentioned, there are a lot of packages for general parallelization. Two commonly used ones are <code>joblib</code> and <code>multiprocessing</code>. <code>joblib</code> looks a bit more like list-recursion and <code>multiprocessing</code> uses functions like <code>.map()</code>.

== <code>multiprocessing</code>/<code>multiprocess</code> ==

The <code>multiprocessing</code> package is actually built into Python, and works in a pretty straightforward manner. It also allows for synchronous vs. asynchronous parallelization, where synchronous means the order of the jobs must be followed, one after the next, and asynchronous means multiple tasks can be executed in small parts in overlapping periods. Note that many statistical uses of parallelization do not need synchronous parallelization.

Since both approaches have the same syntax we’ll be using <code>multiprocess</code> (as it works with <code>ipython</code> and <code>jupyter notebooks</code>). <code>multiprocessing</code> can have problems with <code>ipython</code> and <code>jupyter notebook</code>s. In <code>jupyter notebook</code>s, if you want to use <code>multiprocessing</code> , it needs to be wrapped in <code>if name == "main"</code> to properly fork processes; you can also save your process is as a .py file and import it into your <code>jupyter notebook.</code>

=== Implementation ===

First, you need to identify the number of cores you have:

<syntaxhighlight lang="python">import multiprocess as mp
#Could also using import multiprocessing as mp; the syntax is the same

num_cores = mp.cpu_count()</syntaxhighlight>
==== Synchronous ====

<syntaxhighlight lang="python">%%timeit

with mp.Pool(processes=num_cores-2) as p:
results = p.map(my_simulation, my_k)</syntaxhighlight>
<syntaxhighlight lang="python">209 ms ± 24.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)</syntaxhighlight>
==== Asynchronous ====

<syntaxhighlight lang="python">%%timeit

with mp.Pool(processes=num_cores-2) as p:
results = p.map_async(my_simulation, my_k).get() # not clear why I needed to use .get() to get something returnable for async</syntaxhighlight>
<syntaxhighlight lang="python">48.9 ms ± 3.08 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)</syntaxhighlight>
== <code>joblib</code> ==

The <code>joblib</code> package is pretty straightforward and requires you to structure things in terms of list-recursion with a few wrappers to help the process work as desired. Just like with <code>multiprocessing</code>, the number of cores needs to be computed before proceeding.

=== Implementation ===

<syntaxhighlight lang="python">import joblib

num_cores = joblib.cpu_count()</syntaxhighlight>To use <code>joblib</code> functionality, we can import the functions <code>Parallel</code> and <code>delayed</code>. <code>Parallel</code> creates an instance for parallelization with a set number of jobs based on the core availability. <code>delayed</code> creates deferred function calls to distribute computations across multiple cores; this is the function that allows for simultaneous task execution.

<syntaxhighlight lang="python">from joblib import Parallel, delayed</syntaxhighlight>
We then can move our list-recursive example into our <code>joblib</code> structure. We wrap <code>Parallel(...)()</code> around the inside of the list-recursion and <code>delayed()</code> around the functions that will be repeated multiple times.

<syntaxhighlight lang="python">%% timeit

r = Parallel(n_jobs=num_cores-2)(delayed(my_simulation)(k) for k in my_k)</syntaxhighlight>
<syntaxhighlight lang="python">172 ms ± 23.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)</syntaxhighlight>
=== Add-ons ===

There are multiple add-ons that can be desirable

<ol style="list-style-type: decimal;">
<li>'''Verbosity''' can be added with a single parameter use (as seen below). This will provide more detailed output information.
<syntaxhighlight lang="python">r = Parallel(n_jobs=num_cores-2, verbose=10)(delayed(my_simulation)(k) for k in my_k)</syntaxhighlight></li>
<li>'''Multiple parameters''' can be passed with redefinitions of functions to take groupings. To replace multiple for-loops, often the function <code>itertools.product()</code> is used for the parameters.</li></ol>

= Just in Time compilers + parallelization =

TODO: <code>numba</code> and <code>cython</code>. The idea of these is to compile the function you will be using the first time you use it and then never compile it again (speeding up the actual use of the function). That is then combined with standard parallelization

= Jupyter notebook: <code>ipyparallel</code> =

For those that heavily use jupyter notebooks and nothing else (which this author personally doesn’t encourage), the package <code>ipyparallel</code> nicely works inside jupyter notebooks (but so does many of the other packages above like <code>joblib</code>, <code>multiprocess</code>, etc.). One nice this about this package is the verbosity, etc. works intellegently in the notebook. This package also works with <code>numba.jit</code> and <code>cython</code>.

Before opening your <code>jupyter notebook</code> you’ll want to make sure the extension is enabled - which can be down from the command line with the following command or inside the <code>jupyter notebook</code> extensions tab.

<syntaxhighlight lang="bash">ipcluster nbextension enable</syntaxhighlight>
And then you’ll need to go to the “IPython Clusters” tab in <code>jupyter noteook</code> and start some parallel clusters.

== <code>ipyparallel</code> parallelization ==

<syntaxhighlight lang="python">import ipyparallel as ipp</syntaxhighlight>
For direct execution we define <code>dv</code> and provide it with what it needs…

<syntaxhighlight lang="python">rc = ipp.Client()
dv = rc[:] # direct view</syntaxhighlight>
=== passing things to each cluster ===

<code>ipyparallel</code> provides ways for the user to pass only the necessary items to the the clusters (as to not overload the clusters with too many things).

<ol style="list-style-type: decimal;">
<li>passing packages all together
<syntaxhighlight lang="python">with dv.sync_imports():
from sklearn.linear_model import LinearRegression
# getting dependencies on the clusters</syntaxhighlight></li>
<li>passing variable over:
Below is just an example - we don’t need to pass <code>my_k</code> since <code>dv</code> will directly get it when we wnat to use it.
<syntaxhighlight lang="python">dv["my_k"] = my_k</syntaxhighlight></li>
<li>we can also use the <code>%px</code> magick to run lines of code on all servers. Given I think <code>numpy</code> is cooler as <code>np</code> I actually have to do it this way, not the <code>dv.sync_imports</code> way.
<syntaxhighlight lang="python">%px import numpy as np</syntaxhighlight></li></ol>

=== actual functionality ===

The actual functionality is very similar to <code>multiprocess</code>/<code>multiprocessing</code> with <code>maps</code> and <code>apply</code>.

==== synchronous ====

<syntaxhighlight lang="python">%%timeit
results = dv.map_sync(my_simulation, my_k)</syntaxhighlight>
<syntaxhighlight lang="python">201 ms ± 34.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)</syntaxhighlight>
==== asynchronous ====

<syntaxhighlight lang="python">%%timeit
results = dv.map_async(my_simulation, my_k).get() # not clear why I needed to use .get() to get something returnable for async</syntaxhighlight>
<syntaxhighlight lang="python">209 ms ± 28.1 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)</syntaxhighlight>
= additional topics and resources =

# [https://ray.io/ <code>Ray</code> python package] allows paralleilization across multiple computers
# [https://people.duke.edu/~ccc14/sta-663-2018/notebooks/S14A_Parallel_Programming_Introduction.html Duke’s statistical computing course] provides a set of examples for multiple python parallelization packages |

Parallelization in python

2025-06-25T14:30:32Z

Ecucuzze:

'''This article is still being developed - released earlier to align with Department seminar.'''

In Python, parallelization is a bit more of a “wild west” in terms of the number of parallelization packages available. This wiki will attempt to present minor examples of some common approaches, along with an outline of things to think about in selecting the library you end up using. In general, the process in Python involves:

# defining a function that captures what you want to have parallelized
# calling that function repeatedly in the the parallelization package way (which often looks like a list-recursive call).

There are a few different general frameworks to accomplish this.

# '''General wrappers''' define the cores you’ll be using and then wrap that around the function call in a list-recursive way.
# '''Just in time compilers''', in addition to providing parallelization tools, provide decorators (e.g. <code>@jit</code>) for functions that allow for just in time compilation of the Python function into lower level machine code (like <code>C++</code>).

We will also discuss Python packages that are built specifically to parallelize jobs inside '''Jupyter notebooks'''.

= Non-parallelized: single core example =

We naturally want to compare a parallelization approach with the base single core analysis, which can be done with the code as follows:

<syntaxhighlight lang="python">#base for-loop
import numpy as np
from sklearn.linear_model import LinearRegression

my_k = list(np.arange(14)+1)*20

def my_simulation(k):
"""
estamine the polynomial fit of degree k's MSE for a linear
relationship

Parameters
----------
k : int
degree of polynomial fit

Returns
-------
mse of fit based on 30 points applied to a test set of 10000 points
"""
x = np.random.uniform(size=30,low=-3,high=3)
y = 2*x + 2* np.random.normal(size=30)
y = y.reshape(-1,1)
X_poly = np.concatenate([x.reshape(-1,1)**kk
for kk in np.arange(1,k+1,dtype = int)],
axis = 1)

x0 = np.random.uniform(size=10000,low=-3,high=3)
y0 = 2*x0 + 2*np.random.normal(size=10000)
y0 = y0.reshape(-1,1)
X0_poly = np.concatenate([x0.reshape(-1,1)**kk
for kk in np.arange(1,k+1,dtype = int)],
axis = 1)

my_lm = LinearRegression().fit(X_poly,y)
y0_hat = my_lm.predict(X0_poly)

return(np.mean((y0-y0_hat)**2))
</syntaxhighlight>
<syntaxhighlight lang="python">%%timeit -n3

[my_simulation(k) for k in my_k]</syntaxhighlight>
<syntaxhighlight lang="python">676 ms ± 37.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)</syntaxhighlight>
= General python parallelization packages =

As mentioned there are a lot of package to do general parallelization. Two commonly used ones are <code>joblib</code> and <code>multiprocessing</code>. <code>joblib</code> looks a bit more like list-recursion and <code>multiprocessing</code> uses functions like <code>.map()</code>. <code>multiprocessing</code> can have problems with <code>ipython</code> and <code>jupyter notebook</code>s (though <code>joblib</code> wraps it and <code>multiprocess</code> is just a different fork of <code>multiprocessing</code>).

== <code>multiprocessing</code>/<code>multiprocess</code> ==

The <code>multiprocessing</code> package is actually built into python, and works in a pretty straightforward manner. It also allows for synchronous vs asynchronous parallelization, where synchronous means the order of the jobs must be followed and asynchronous where the order of jobs need not be followed. Many statistical uses of parallelization does not need synchronous parallelization.

There since both approaches have the same syntax we’ll be using <code>multiprocess</code> (as it works with <code>ipython</code> and <code>jupyter notebooks</code>).

=== parallelization ===

First, you need to identify the number of cores you have:

<syntaxhighlight lang="python">import multiprocess as mp
#import multiprocessing as mp

num_cores = mp.cpu_count()</syntaxhighlight>
==== synchronous ====

<syntaxhighlight lang="python">%%timeit

with mp.Pool(processes=num_cores-2) as p:
results = p.map(my_simulation, my_k)</syntaxhighlight>
<syntaxhighlight lang="python">209 ms ± 24.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)</syntaxhighlight>
==== asynchronous ====

<syntaxhighlight lang="python">%%timeit

with mp.Pool(processes=num_cores-2) as p:
results = p.map_async(my_simulation, my_k).get() # not clear why I needed to use .get() to get something returnable for async</syntaxhighlight>
<syntaxhighlight lang="python">48.9 ms ± 3.08 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)</syntaxhighlight>
== <code>joblib</code> ==

The <code>joblib</code> package is pretty straightforward and requires you to structure things in terms of list-recursion with a few wrappers to help the process work as desired.

=== setup ===

<syntaxhighlight lang="python">import joblib

num_cores = joblib.cpu_count()</syntaxhighlight>
=== implimentation ===

To use <code>joblib</code> functionality we can import the functions <code>Parallel</code> and <code>delayed</code>.

<syntaxhighlight lang="python">from joblib import Parallel, delayed</syntaxhighlight>
We then can move our list-recursive example into our <code>joblib</code> structure. We wrap <code>Parallel(...)()</code> around the old of the inside of the list-recursion and <code>delayed()</code> around the functions that will be repeated multiple times.

<syntaxhighlight lang="python">%% timeit

r = Parallel(n_jobs=num_cores-2)(delayed(my_simulation)(k) for k in my_k)</syntaxhighlight>
<syntaxhighlight lang="python">172 ms ± 23.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)</syntaxhighlight>
=== Add-ons ===

There are multiple add-ons that can be desirable

<ol style="list-style-type: decimal;">
<li>'''verbosity''' can be added with a single parameter use (as seen below) - the large the value, the more messages:
<syntaxhighlight lang="python">r = Parallel(n_jobs=num_cores-2, verbose=10)(delayed(my_simulation)(k) for k in my_k)</syntaxhighlight></li>
<li>'''multiple parameters''' can be passed with redefinitions of functions to take groupings. To replace multiple for-loops often the function <code>itertools.product()</code> is used for the parameters.</li></ol>

= Just in Time compilers + parallelization =

TODO: <code>numba</code> and <code>cython</code>. The idea of these is to compile the function you will be using the first time you use it and then never compile it again (speeding up the actual use of the function). That is then combined with standard parallelization

= Jupyter notebook: <code>ipyparallel</code> =

For those that heavily use jupyter notebooks and nothing else (which this author personally doesn’t encourage), the package <code>ipyparallel</code> nicely works inside jupyter notebooks (but so does many of the other packages above like <code>joblib</code>, <code>multiprocess</code>, etc.). One nice this about this package is the verbosity, etc. works intellegently in the notebook. This package also works with <code>numba.jit</code> and <code>cython</code>.

Before opening your <code>jupyter notebook</code> you’ll want to make sure the extension is enabled - which can be down from the command line with the following command or inside the <code>jupyter notebook</code> extensions tab.

<syntaxhighlight lang="bash">ipcluster nbextension enable</syntaxhighlight>
And then you’ll need to go to the “IPython Clusters” tab in <code>jupyter noteook</code> and start some parallel clusters.

== <code>ipyparallel</code> parallelization ==

<syntaxhighlight lang="python">import ipyparallel as ipp</syntaxhighlight>
For direct execution we define <code>dv</code> and provide it with what it needs…

<syntaxhighlight lang="python">rc = ipp.Client()
dv = rc[:] # direct view</syntaxhighlight>
=== passing things to each cluster ===

<code>ipyparallel</code> provides ways for the user to pass only the necessary items to the the clusters (as to not overload the clusters with too many things).

<ol style="list-style-type: decimal;">
<li>passing packages all together
<syntaxhighlight lang="python">with dv.sync_imports():
from sklearn.linear_model import LinearRegression
# getting dependencies on the clusters</syntaxhighlight></li>
<li>passing variable over:
Below is just an example - we don’t need to pass <code>my_k</code> since <code>dv</code> will directly get it when we wnat to use it.
<syntaxhighlight lang="python">dv["my_k"] = my_k</syntaxhighlight></li>
<li>we can also use the <code>%px</code> magick to run lines of code on all servers. Given I think <code>numpy</code> is cooler as <code>np</code> I actually have to do it this way, not the <code>dv.sync_imports</code> way.
<syntaxhighlight lang="python">%px import numpy as np</syntaxhighlight></li></ol>

=== actual functionality ===

The actual functionality is very similar to <code>multiprocess</code>/<code>multiprocessing</code> with <code>maps</code> and <code>apply</code>.

==== synchronous ====

<syntaxhighlight lang="python">%%timeit
results = dv.map_sync(my_simulation, my_k)</syntaxhighlight>
<syntaxhighlight lang="python">201 ms ± 34.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)</syntaxhighlight>
==== asynchronous ====

<syntaxhighlight lang="python">%%timeit
results = dv.map_async(my_simulation, my_k).get() # not clear why I needed to use .get() to get something returnable for async</syntaxhighlight>
<syntaxhighlight lang="python">209 ms ± 28.1 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)</syntaxhighlight>
= additional topics and resources =

# [https://ray.io/ <code>Ray</code> python package] allows paralleilization across multiple computers
# [https://people.duke.edu/~ccc14/sta-663-2018/notebooks/S14A_Parallel_Programming_Introduction.html Duke’s statistical computing course] provides a set of examples for multiple python parallelization packages |

Parallelization in python

2025-06-24T15:35:40Z

Ecucuzze:

'''This article is still being developed - released earlier to align with Department seminar.'''

In python, parallelization is a bit more of a “wild west” in terms of the number of packages that can help you parallelize your processes (This wiki will attempt to present minor examples of some along with an outline of things to think about in selecting the library you end up using. In general, the process in python involves

# defining a function that captures what you want to have parallelized
# call that function repeatedly in the the parallelization package way (which often looks like a list-recursive call).

There are a few different general frameworks to accomplish this.

# '''general wrappers''', define the cores you’ll be using and then wrap that around the function call in a list-recursive way
# '''just in time compilers''', in addition to providing parallelization tools these packages provide decorators (e.g. <code>@jit</code>) for functions that allow for just in time compilation of the python function into lower level machine code (like <code>C++</code>)

We will discuss a python package that is build specifically to parallelize jobs inside '''jupyter notebooks'''.

= Non-parallelized: single core example =

We naturally want to compare a parallelization approach with the base single core analysis, which can be done with the code as follows:

<syntaxhighlight lang="python">#base for-loop
import numpy as np
from sklearn.linear_model import LinearRegression

my_k = list(np.arange(14)+1)*20

def my_simulation(k):
"""
estamine the polynomial fit of degree k's MSE for a linear
relationship

Parameters
----------
k : int
degree of polynomial fit

Returns
-------
mse of fit based on 30 points applied to a test set of 10000 points
"""
x = np.random.uniform(size=30,low=-3,high=3)
y = 2*x + 2* np.random.normal(size=30)
y = y.reshape(-1,1)
X_poly = np.concatenate([x.reshape(-1,1)**kk
for kk in np.arange(1,k+1,dtype = int)],
axis = 1)

x0 = np.random.uniform(size=10000,low=-3,high=3)
y0 = 2*x0 + 2*np.random.normal(size=10000)
y0 = y0.reshape(-1,1)
X0_poly = np.concatenate([x0.reshape(-1,1)**kk
for kk in np.arange(1,k+1,dtype = int)],
axis = 1)

my_lm = LinearRegression().fit(X_poly,y)
y0_hat = my_lm.predict(X0_poly)

return(np.mean((y0-y0_hat)**2))
</syntaxhighlight>
<syntaxhighlight lang="python">%%timeit -n3

[my_simulation(k) for k in my_k]</syntaxhighlight>
<syntaxhighlight lang="python">676 ms ± 37.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)</syntaxhighlight>
= General python parallelization packages =

As mentioned there are a lot of package to do general parallelization. Two commonly used ones are <code>joblib</code> and <code>multiprocessing</code>. <code>joblib</code> looks a bit more like list-recursion and <code>multiprocessing</code> uses functions like <code>.map()</code>. <code>multiprocessing</code> can have problems with <code>ipython</code> and <code>jupyter notebook</code>s (though <code>joblib</code> wraps it and <code>multiprocess</code> is just a different fork of <code>multiprocessing</code>).

== <code>multiprocessing</code>/<code>multiprocess</code> ==

The <code>multiprocessing</code> package is actually built into python, and works in a pretty straightforward manner. It also allows for synchronous vs asynchronous parallelization, where synchronous means the order of the jobs must be followed and asynchronous where the order of jobs need not be followed. Many statistical uses of parallelization does not need synchronous parallelization.

There since both approaches have the same syntax we’ll be using <code>multiprocess</code> (as it works with <code>ipython</code> and <code>jupyter notebooks</code>).

=== parallelization ===

First, you need to identify the number of cores you have:

<syntaxhighlight lang="python">import multiprocess as mp
#import multiprocessing as mp

num_cores = mp.cpu_count()</syntaxhighlight>
==== synchronous ====

<syntaxhighlight lang="python">%%timeit

with mp.Pool(processes=num_cores-2) as p:
results = p.map(my_simulation, my_k)</syntaxhighlight>
<syntaxhighlight lang="python">209 ms ± 24.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)</syntaxhighlight>
==== asynchronous ====

<syntaxhighlight lang="python">%%timeit

with mp.Pool(processes=num_cores-2) as p:
results = p.map_async(my_simulation, my_k).get() # not clear why I needed to use .get() to get something returnable for async</syntaxhighlight>
<syntaxhighlight lang="python">48.9 ms ± 3.08 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)</syntaxhighlight>
== <code>joblib</code> ==

The <code>joblib</code> package is pretty straightforward and requires you to structure things in terms of list-recursion with a few wrappers to help the process work as desired.

=== setup ===

<syntaxhighlight lang="python">import joblib

num_cores = joblib.cpu_count()</syntaxhighlight>
=== implimentation ===

To use <code>joblib</code> functionality we can import the functions <code>Parallel</code> and <code>delayed</code>.

<syntaxhighlight lang="python">from joblib import Parallel, delayed</syntaxhighlight>
We then can move our list-recursive example into our <code>joblib</code> structure. We wrap <code>Parallel(...)()</code> around the old of the inside of the list-recursion and <code>delayed()</code> around the functions that will be repeated multiple times.

<syntaxhighlight lang="python">%% timeit

r = Parallel(n_jobs=num_cores-2)(delayed(my_simulation)(k) for k in my_k)</syntaxhighlight>
<syntaxhighlight lang="python">172 ms ± 23.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)</syntaxhighlight>
=== Add-ons ===

There are multiple add-ons that can be desirable

<ol style="list-style-type: decimal;">
<li>'''verbosity''' can be added with a single parameter use (as seen below) - the large the value, the more messages:
<syntaxhighlight lang="python">r = Parallel(n_jobs=num_cores-2, verbose=10)(delayed(my_simulation)(k) for k in my_k)</syntaxhighlight></li>
<li>'''multiple parameters''' can be passed with redefinitions of functions to take groupings. To replace multiple for-loops often the function <code>itertools.product()</code> is used for the parameters.</li></ol>

= Just in Time compilers + parallelization =

TODO: <code>numba</code> and <code>cython</code>. The idea of these is to compile the function you will be using the first time you use it and then never compile it again (speeding up the actual use of the function). That is then combined with standard parallelization

= Jupyter notebook: <code>ipyparallel</code> =

For those that heavily use jupyter notebooks and nothing else (which this author personally doesn’t encourage), the package <code>ipyparallel</code> nicely works inside jupyter notebooks (but so does many of the other packages above like <code>joblib</code>, <code>multiprocess</code>, etc.). One nice this about this package is the verbosity, etc. works intellegently in the notebook. This package also works with <code>numba.jit</code> and <code>cython</code>.

Before opening your <code>jupyter notebook</code> you’ll want to make sure the extension is enabled - which can be down from the command line with the following command or inside the <code>jupyter notebook</code> extensions tab.

<syntaxhighlight lang="bash">ipcluster nbextension enable</syntaxhighlight>
And then you’ll need to go to the “IPython Clusters” tab in <code>jupyter noteook</code> and start some parallel clusters.

== <code>ipyparallel</code> parallelization ==

<syntaxhighlight lang="python">import ipyparallel as ipp</syntaxhighlight>
For direct execution we define <code>dv</code> and provide it with what it needs…

<syntaxhighlight lang="python">rc = ipp.Client()
dv = rc[:] # direct view</syntaxhighlight>
=== passing things to each cluster ===

<code>ipyparallel</code> provides ways for the user to pass only the necessary items to the the clusters (as to not overload the clusters with too many things).

<ol style="list-style-type: decimal;">
<li>passing packages all together
<syntaxhighlight lang="python">with dv.sync_imports():
from sklearn.linear_model import LinearRegression
# getting dependencies on the clusters</syntaxhighlight></li>
<li>passing variable over:
Below is just an example - we don’t need to pass <code>my_k</code> since <code>dv</code> will directly get it when we wnat to use it.
<syntaxhighlight lang="python">dv["my_k"] = my_k</syntaxhighlight></li>
<li>we can also use the <code>%px</code> magick to run lines of code on all servers. Given I think <code>numpy</code> is cooler as <code>np</code> I actually have to do it this way, not the <code>dv.sync_imports</code> way.
<syntaxhighlight lang="python">%px import numpy as np</syntaxhighlight></li></ol>

=== actual functionality ===

The actual functionality is very similar to <code>multiprocess</code>/<code>multiprocessing</code> with <code>maps</code> and <code>apply</code>.

==== synchronous ====

<syntaxhighlight lang="python">%%timeit
results = dv.map_sync(my_simulation, my_k)</syntaxhighlight>
<syntaxhighlight lang="python">201 ms ± 34.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)</syntaxhighlight>
==== asynchronous ====

<syntaxhighlight lang="python">%%timeit
results = dv.map_async(my_simulation, my_k).get() # not clear why I needed to use .get() to get something returnable for async</syntaxhighlight>
<syntaxhighlight lang="python">209 ms ± 28.1 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)</syntaxhighlight>
= additional topics and resources =

# [https://ray.io/ <code>Ray</code> python package] allows paralleilization across multiple computers
# [https://people.duke.edu/~ccc14/sta-663-2018/notebooks/S14A_Parallel_Programming_Introduction.html Duke’s statistical computing course] provides a set of examples for multiple python parallelization packages |