Slurm

Slurm is a job management system that deals with job prioritization and the allocation of resources. On high-performance computers slurm helps make sure jobs are run as rapidly as possible. Because this system manages how you are able to use the resources and when your job is run, it requires you to know additional information before you submit your job. Additionally, given that the PSC uses slurm, one should it as a tool for when you have a really good understanding of your code and have attempted to make it efficient, especially since you’re often constrained by the number of core-hours you use. To submit jobs you’ll need to have 1. an understanding of the time it will take to run your job (actually an over-estimate of such things) [see R and python references on approaches to estimate this - to do.] 2. and (to a lesser extent) the amount of storage and RAM you will be using for the job (this varies across what type of access you have at PSC).

Slurm’s most powerful feature: parallelization

The most powerful feature of slurm is probably its ability to run jobs in parallel outside of the programming language [see articles for inside language parallelization in R and python]. Parallelization is often useful in statistical analysis when the analysis can be broken up into many subparts (e.g. parameter tuning, data processing, simulations, etc.). When using a system with slurm, it’s smart to parallelize with slurm (outside of the programming language) if not too costly so that slurm can schedule smaller (in core-hour) jobs and get them actually run more quickly.

Basic / commonly used commands

Because slurm decides when your code will run, it is important to know what resources are being used when you’re about to run a job. Similar to the department server, conscientious usage of the servers with slurm is the best approach (this varies across what type of access you have at PSC). Additionally, given that slurm can constrain the number of core hours you use, it’s useful to know how to cancel your jobs if something is not processing correctly.

Commmand	Description	Unix similarity
squeue	get a complete list of all jobs running	top
squeue -u username	all jobs running for just a single user	top -u username
sinfo	provide information about the state of each node (look for idle nodes)
scancel	cancel jobs	slightly similar to canceling a job with a terminal multiplexer
scancel -u username	cancel all jobs for a single user (only can do with yourself)
scancel _JobID_	cancel job with id: _JobID__
sbatch _bash_script_	submit batch request with a .sh bash script	slightly similar to running a job in a terminal multiplexer

Additional documentation: slurm.schedmd, osc.edu, local astro-PSC docs: vera-doc.psc.edu

Some additional useful tools

Based on feedback we wanted to give you some additional references to getting useful information about your jobs:

1. How to see the number of jobs you have Running, Pending, Completed, and Failed: slackoverflow
2. How to understand the amount of core hours you'ved used:
   To see the time used for each individual job:

   sacct -S2020-08-09 -u username -ojobid,start,end,alloccpu,cputime | column -t

   where:
   - ojobid: job ID
   - alloccpu: number of cpus allocated
   - cputime: amount of cpu time used
   
   To get the total number of core-hours used (summing up column 5) run:

   sacct -S2020-08-09 -u username -ojobid,start,end,alloccpu,cputime | column -t | awk '{print $5}' | awk '{s+=$1}END{print s}'

3. How to transfer files: see Linux on the server commands (ssh, sftp, GUI based transferring)

 

Example of how execute jobs

To actually execute a job it’s common practice to describe what you’d like to do in a bash file. This helps keep all the specifications about the job request in one place.

An example bash script batch-script.sh for running a set of .R files (with a few inputs) can be seen below:

#!/bin/bash
#SBATCH -t 01:00:00
#SBATCH -N 1
#SBATCH --ntasks-per-node 1
#SBATCH --mail-user=andrew_id@andrew.cmu.edu
#SBATCH --mail-type=END

set -x

Rscript code.R

To run this file we’d run (on the server):

sbatch batch-script.sh

The following structures should be observed:

1. #!/bin/bash [Required], this reminds the compiler that this is a bash script
2. SBATCH -t 01:00:00 [Required], this tells slurm that we expect a maximum of 1 hour to be need to run each job (if the job is running past 1 hour it will cancel - so be conservative in this estimate!)
3. SBATCH -N 1 [Required], this tells slurm that we only need 1 nodes for each job (a node contains multiple cores)
4. SBATCH --ntasks-per-node 1, this tells slurm we only need 1 core for each job (in this example we are parallelizing with slum)
5. SBATCH --mail-user=andrew_id@andrew.cmu.edu, this makes slurm send you an email when all your jobs from this file is done (so you don’t have to keep logging in to see if the job failed / completed)
6. SBATCH --mail-type=END, defines when you’ll get the email (at the end of all all jobs)
7. set -x, stores console lines from job runs into a log file (good for debugging). Without a line above saying SBATCH -o log_file_$SLURM_JOB_ID.tx it will select the file’s name itself. 
8. Rscript code.R $SLURM_ARRAY_TASK_ID 100, the actual job we’re running. In this case we’re running an R file with input parameters associated with our job array number in line 4 and a second parameter 100. For python users, this could easily be replaced with ipython code.py $SLURM_ARRAY_TASK_ID 100.

Here is the filled-out content for your wiki entry, using MediaWiki syntax and keeping the tone consistent with your existing documentation.

Using SLURM to parallelize workflows

Manual submission of multiple jobs

If you have a handful of distinct jobs (e.g., running a model on three different datasets: A, B, and C), the simplest method is to submit them one by one. While you could type sbatch batch-script-A.sh, sbatch batch-script-B.sh, etc., into the console manually, you can also put these commands into a simple bash shell script (often called a "driver" or "launcher" script).

#!/bin/bash

# launcher.sh

# Submit three distinct jobs

sbatch batch-script-A.sh
sbatch batch-script-B.sh
sbatch batch-script-C.sh

When you run bash launcher.sh, Slurm will accept all three jobs, give them unique Job IDs, and queue them independently. They may run simultaneously or sequentially depending on available resources.

Automated submission of multiple jobs

For larger batches where the code is identical but the input parameters change, writing individual script files is inefficient. You can automate the generation and submission of these jobs.

Method 1: Generating jobs on the fly (No external template)

You can use a loop in a bash script to submit jobs dynamically using the `--wrap` flag (for simple one-liners) or by piping a "Here Document" (Heredoc) into `sbatch`. The Heredoc method is preferred for complex scripts as it allows you to inject bash variables directly into the Slurm script instructions.

#!/bin/bash

# loop_submit.sh

# Loop through parameters 1 to 10

for i in {1..10}
do
echo "Submitting job for parameter $i..."

```
# We pipe the script configuration directly into sbatch
# $i is expanded automatically inside the EOF block
sbatch <<EOF

```

#!/bin/bash
#SBATCH -t 01:00:00
#SBATCH -N 1
#SBATCH --ntasks-per-node 1
#SBATCH -o "output_param_{i}"

set -x
echo "Running iteration $i"
Rscript code.R $i 100
EOF

done

Method 2: Using a template file with Exported Variables

Instead of modifying the script file text (e.g., using sed), a cleaner, more robust approach is to pass environment variables directly to the job using the --export flag. This allows you to use a single static .sb file for all jobs.

First, create your generic Slurm script (template.sb). Notice we use the variable $MY_INPUT inside the script, but we never define it there.

# template.sb
#!/bin/bash

#SBATCH -t 01:00:00 #SBATCH -N 1 #SBATCH --ntasks-per-node 1

# We expect MY_INPUT to be passed in from sbatch
echo "Running analysis on config: $MY_INPUT"

Rscript code.R $MY_INPUT 100

Then, create a submission loop. We use --export=ALL,MY_INPUT="value" to pass the current loop variable into the Slurm job's environment.

# submit_export.sh
#!/bin/bash

# A list of imaginary config names
configs="CONFIG_A CONFIG_B CONFIG_C"

for cfg in $configs do

# Strip 'CONFIG_' prefix for a cleaner job name
jobname="${cfg#CONFIG_}"

echo "Submitting job for $cfg with job name $jobname"

# Submit the job
# --export=ALL passes current env vars
# MY_INPUT="$cfg" sets the specific variable our template needs
sbatch --job-name="$jobname" --output="log_${jobname}.out" --export=ALL,MY_INPUT="$cfg" template.sb
done

Job Arrays

While the looping methods above work, they are frowned upon for large numbers of jobs (e.g., 1,000 simulations). Submitting 1,000 individual jobs stresses the scheduler, clogs the queue for other users, and makes it difficult for you to manage your own jobs (e.g., you cannot easily cancel all of them at once).

Job Arrays are the preferred solution. They bundle many identical jobs into a single job ID with multiple "indices."

#!/bin/bash
#SBATCH -t 01:00:00
#SBATCH -N 1
#SBATCH --ntasks-per-node 1
#SBATCH --array=1-100       # <--- This is the key line
#SBATCH -o array_job_%A_%a.out # %A is Job ID, %a is Array Index

set -x

# Slurm automatically provides the variable $SLURM_ARRAY_TASK_ID

# containing the index number (1, 2, ... 100) for that specific task.

echo "This is task number $SLURM_ARRAY_TASK_ID"
Rscript code.R $SLURM_ARRAY_TASK_ID 100

To run this, you simply `sbatch` this single file. Slurm will spawn 100 tasks. If you need to cancel them all, you only need to cancel the one main Job ID.

Within job parallelization

Sometimes you do not want to spread work across many different allocations, but rather utilize multiple CPUs/GPUs on a single allocation to speed up one large job.

Scenario 1: Parallel Tasks with srun (Process Parallelism)

If you want to run multiple instances of a program simultaneously (process-based parallelism) rather than one program using multiple cores, you should use --ntasks combined with srun.

When you prefix a command with srun inside a script, Slurm launches that command across all the tasks (processors) you requested in your header. This is useful for MPI applications or for running multiple copies of a script where each copy identifies its own work based on its "Task ID."

#!/bin/bash 
#SBATCH -N 1 
#SBATCH --ntasks=4 # Request 4 specific tasks (processes) 
#SBATCH --cpus-per-task=2 # Request 2 CPUS per task 
#SBATCH --mem-per-cpu=2G # Request 2 GB of memory per CPU, so 4 GB per task
#SBATCH -t 01:00:00

# We use srun to launch the script.
# Because we requested ntasks=4, srun will launch 4 simultaneous copies of 'code.py'. Each copy will get 2 CPUs and 4 GB of memory
srun python code.py

Note: In this scenario, your Python/R script needs to know which "instance" it is so it doesn't just do the exact same work 4 times. You can access the environment variable SLURM_PROCID (which will be 0, 1, 2, or 3) inside your code to determine which chunk of data that specific instance should process. While you can request CPUs and memory per task, any GPUs requested via --gres:gpu=[#] will be shared across all tasks

Scenario 2: Chaining background tasks (The "&" operator)

If you have several small, independent steps that need to run within the same allocation, you can run them in the background using the bash ampersand `&` operator. You must use the `wait` command at the end; otherwise, the script will finish and kill the background jobs immediately.

#!/bin/bash
#SBATCH -N 1
#SBATCH -t 01:00:00

# Run first python script in the background

python process_part_A.py > output_A.log &

# Run second python script in the background

python process_part_B.py > output_B.log &

# IMPORTANT: Wait for all background processes to finish before exiting

wait

echo "Both parts A and B have finished."

---

SLURM Demo

For a hands on demonstration using SLURM, click here.