View source for Parallelization on PSC/Bridges-2


This document aims to outline the essential steps for getting up-and-running as quickly as possible on the Pittsburgh Supercomputing Center (PSC) and Bridges-2, with a particular focus on running parallel computation in Python and R.

These notes should be treated as a specific application-focused complement to the&nbsp;[https://portal.xsede.org/psc-bridges-2#running-sbatch official documentation]&nbsp;rather than a substitute.&nbsp;This document is a work in progress. Please feel free to contribute to it!


== '''Account Setup''' ==

XSEDE is now discontinued and the system has transitioned to ACCESS.&nbsp;

'''TBD'''

== '''Running your first job''' ==

=== Logging into Bridges2/PSC via&nbsp;<code>ssh</code> ===

Start by&nbsp;<code>ssh</code>ing into the bridges2 server at port 2222:
<pre>ssh -p 2222 <username>@bridges2.psc.edu</pre>

You will be prompted for your XSEDE credentials. Use your XSEDE User Portal password.


=== Creating a slurm job batch script ===

Bridges-2 uses a job-scheduling system called "slurm", providing a way for users like ourselves to request specific resources for our compute jobs and automatically launch jobs once said resources have become available.

Jobs are submitted using the&nbsp;<code>sbatch</code>&nbsp;command along with a&nbsp;'''batch script'''&nbsp;which we must create. A batch script may look like the following.
<pre dir="auto">#!/bin/bash
#SBATCH -p EM
#SBATCH -N 1
#SBATCH -n 24
#SBATCH --mem 5GB
#SBATCH -t 00:01:00
#SBATCH --mail-type=ALL

# Load Anaconda
module load anaconda3
conda activate

python parallel.py</pre>

The above script requests the Extreme Memory (EM) partition, with one node, 24 cores, 5GB of memory, for 1 minute, and sends an email to the job runner once the script begins/finishes/fails/etc. This script then loads a&nbsp;<code>conda</code>&nbsp;environment and runs a python script called&nbsp;<code>parallel.py</code>&nbsp;which we will discuss in the&nbsp;[https://github.com/WannabeSmith/PSC-Bridges-resources#python python]&nbsp;example.

More generally, these batch scripts take the following form.
<pre>#!/bin/bash

### REQUEST RESOURCES ###
#SBATCH -arg1 argument1
#SBATCH -arg2 argument2 
# .
# .
# .
#SBATCH -arg3 argument3

### LOAD MODULES ###
module load some_module

### RUN SCRIPTS ###
python myscript.py
Rscript myscript.R</pre>

The area under "REQUEST RESOURCES" is where we request specific resources such as the partition of your allocation to run on (e.g. RM or EM for regular memory or extreme memory, respectively), the amount of compute time, the number of nodes, the number of cores, and so on.

Some common options are given in the following table.

{| border="1" cellpadding="1" cellspacing="1" style="width: 686px;"
|-
! scope="col" style="width: 194px;" | Argument
! scope="col" style="width: 195px;" | Description
! scope="col" style="width: 275px;" | Example
|-
| style="width: 194px;" | <pre>-p <partition></pre>

| style="width: 195px;" | Partition
| style="width: 275px;" | <pre>-p EM</pre>

|-
| style="width: 194px;" | <pre>-t <HH:MM:SS></pre>

| style="width: 195px;" | Amount of compute time
| style="width: 275px;" | <pre><code>-t 00:01:00</code> 
</pre>

(equivalent to 1 hour)

|-
| style="width: 194px;" | <pre>-N <nodes></pre>

| style="width: 195px;" | Number of nodes
| style="width: 275px;" | <pre>-N 1</pre>

|-
| style="width: 194px;" | <pre>-n <cores></pre>

| style="width: 195px;" | Number of total cores
| style="width: 275px;" | <pre>-n 25</pre>

|-
| style="width: 194px;" | <pre>--mem=<memory in GB></pre>

| style="width: 195px;" | Amount of RAM in GB
| style="width: 275px;" | <pre>--mem=10GB</pre>

|-
| style="width: 194px;" | <pre>--mail-type=<type></pre>

| style="width: 195px;" | Type of mail to receive (one of&nbsp;<code>ALL</code>,&nbsp;<code>FAIL</code>,&nbsp;<code>START</code>,&nbsp;<code>END</code>)
| style="width: 275px;" | <pre><code>--mail-type=ALL</code>&nbsp;
</pre>

or&nbsp;
<pre><code>--mail-type=FAIL</code></pre>

|}

For a comprehensive list, see&nbsp;[https://portal.xsede.org/psc-bridges-2#running-sbatch https://portal.xsede.org/psc-bridges-2#running-sbatch].

"LOAD MODULES" is where you load the required modules to run your scripts. For example, loading python often requires a conda environment. To see the list of all possible modules, you can run&nbsp;<code>module avail</code>.

"RUN SCRIPTS" is where you run your python/R/julia/etc. scripts. If executed correctly, these scripts will have access to the resources requested by the batch script.

Let's see some of these ideas used in a parallel computing example.

=== Parallel computation ===

In this section, we will perform parallel computations using the examples in&nbsp;[https://github.com/WannabeSmith/PSC-Bridges-resources/blob/main/examples examples]&nbsp;with implementations in both Python and R, alongside an appropriately designed slurm job batch script.

Clone this repository wherever you keep your projects on bridges2.psc.edu.
<pre dir="auto"># After logging into bridges2.psc.edu.
# In my/projects/folder

git clone https://github.com/WannabeSmith/PSC-Bridges-resources.git</pre>

Using either Python or R, we will be running a function&nbsp;<code>slow_successor(i)</code>&nbsp;which simply returns&nbsp;<code>i+1</code>&nbsp;but with a time delay. We will run this function on the integers 0 through 99, using parallel and non-parallel implementations to see the difference in computation time.

To use Python, continue on to the next section. To use R, jump to&nbsp;[https://github.com/WannabeSmith/PSC-Bridges-resources#r R].


==== Python ====

We will recreate the example from&nbsp;[https://github.com/WannabeSmith/PSC-Bridges-resources/blob/main/examples/python/python_parallel examples/python/python_parallel].

Enter the folder&nbsp;<code>examples/python/python_parallel</code>.
<pre>cd examples/python/python_parallel</pre>

We will be running a rather simple function&nbsp;<code>slow_successor(i)</code>&nbsp;which simply returns&nbsp;<code>i+1</code>&nbsp;but with a time delay. We will run this function on the integers 0 through 99, using parallel and non-parallel implementations to see the difference in computation time.

Our slurm batch script&nbsp;<code>jobscript.job</code>&nbsp;is written as the following.
<pre># jobscript.job

#!/bin/bash
#SBATCH -p EM
#SBATCH -N 1
#SBATCH -n 24
#SBATCH --mem 5GB
#SBATCH -t 00:01:00
#SBATCH --mail-type=ALL

# Load Anaconda
module load anaconda3
conda activate

python parallel.py</pre>

The above batch script requests several resources, loads a&nbsp;<code>conda</code>&nbsp;environment, and runs a python file&nbsp;<code>parallel.py</code>. This file&nbsp;<code>parallel.py</code>&nbsp;is written as follows.
<pre># parallel.py

import multiprocessing
import time


def slow_successor(i):
    """
    An intentionally slow function that we wish to parallelize
    """
    time.sleep(0.1)
    return i + 1


# We will run `slow_successor` on the numbers 0 through num_iterations-1.
num_iterations = 100

# Non-parallel implementation:
start_time = time.time()
result = list(map(slow_successor, range(num_iterations)))
end_time = time.time()

print("Non-parallel compute time: " + str(end_time - start_time) + " seconds.")


# Parallel implementation:
n_cores = multiprocessing.cpu_count()
start_time = time.time()
with multiprocessing.Pool(n_cores) as p:
    result = p.map(slow_successor, range(num_iterations))
end_time = time.time()

print(
    "Parallel compute time on "
    + str(n_cores)
    + " cores: "
    + str(end_time - start_time)
    + str(" seconds")
)</pre>

We can now run&nbsp;<code>parallel.py</code>&nbsp;using the resources allocated via&nbsp;<code>jobscript.job</code>&nbsp;with the&nbsp;<code>sbatch</code>&nbsp;command.
<pre>sbatch jobscript.job</pre>

Running&nbsp;<code>sbatch</code>&nbsp;will submit a job with particular job ID (e.g. 1234567). The output from the job will then be written to&nbsp;<code>slurm-<jobid>.out</code>&nbsp;(e.g.&nbsp;<code>slurm-1234567.out</code>). The output from&nbsp;<code>jobscript.job</code>&nbsp;should look something like
<pre>Non-parallel compute time: 10.018450498580933 seconds.
Parallel compute time on 96 cores: 0.3378734588623047 seconds</pre>

==== R ====

We will recreate the example from&nbsp;[https://github.com/WannabeSmith/PSC-Bridges-resources/blob/main/examples/R/R_parallel examples/R/R_parallel].

Enter the folder&nbsp;<code>examples/R/R_parallel</code>.
<pre>cd examples/R/R_parallel</pre>

Our slurm batch script&nbsp;<code>jobscript.job</code>&nbsp;is written as the following.
<pre># jobscript.job

#!/bin/bash
#SBATCH -p EM
#SBATCH -N 1
#SBATCH -n 24
#SBATCH --mem 5GB
#SBATCH -t 00:01:00
#SBATCH --mail-type=ALL

Rscript parallel.R</pre>

The above batch script requests several resources, and runs an R script&nbsp;<code>parallel.R</code>. This file&nbsp;<code>parallel.R</code>&nbsp;is written as follows.
<pre># parallel.R

library(parallel)


# An intentionally slow function that we wish to parallelize
slow_successor <- function(i)
{
  Sys.sleep(0.1)
  return(i + 1)
}


# We will run `slow_successor` on the numbers 0 through num_iterations-1.
num_iterations <- 100


# Non-parallel implementation:
start_time <- Sys.time()
result <- unlist(lapply(1:num_iterations, slow_successor))
end_time <- Sys.time()

time_elapsed <- difftime(end_time, start_time, units = "secs")

print(paste("Non-parallel compute time:", time_elapsed, "seconds."))


# Parallel implementation:
n_cores <- detectCores()

start_time <- Sys.time()
result <- unlist(mclapply(0:(num_iterations-1), slow_successor, mc.cores=n_cores))
end_time <- Sys.time()

time_elapsed <- difftime(end_time, start_time, units = "secs")

print(paste("Parallel compute time on", n_cores, "cores:", time_elapsed, "seconds."))
</pre>


==== Running the Python or R script via&nbsp;<code>sbatch</code> ====

We can now run our Python or R script using the resources allocated via&nbsp;<code>jobscript.job</code>&nbsp;with the&nbsp;<code>sbatch</code>&nbsp;command.
<pre>sbatch jobscript.job</pre>

Running&nbsp;<code>sbatch</code>&nbsp;will submit a job with particular job ID (e.g. 1234567). The output from the job will then be written to&nbsp;<code>slurm-<jobid>.out</code>&nbsp;(e.g.&nbsp;<code>slurm-1234567.out</code>). The output from&nbsp;<code>jobscript.job</code>&nbsp;should look something like
<pre>Non-parallel compute time: 10.018450498580933 seconds.
Parallel compute time on 96 cores: 0.3378734588623047 seconds</pre>