Slurm

Slurm is a job management system that deals with job prioritization and the allocation of resources. On high-performance computers slurm helps make sure jobs are run as rapidly as possible. Because this system manages how you are able to use the resources and when your job is run, it requires you to know additional information before you submit your job. Additionally, given that the PSC uses slurm, one should it as a tool for when you have a really good understanding of your code and have attempted to make it efficient, especially since you’re often constrained by the number of core-hours you use. To submit jobs you’ll need to have 1. an understanding of the time it will take to run your job (actually an over-estimate of such things) [see R and python references on approaches to estimate this - to do.] 2. and (to a lesser extent) the amount of storage and RAM you will be using for the job (this varies across what type of access you have at PSC).

Slurm’s most powerful feature: parallelization

The most powerful feature of slurm is probably its ability to run jobs in parallel outside of the programming language [see articles for inside language parallelization in R and python]. Parallelization is often useful in statistical analysis when the analysis can be broken up into many subparts (e.g. parameter tuning, data processing, simulations, etc.). When using a system with slurm, it’s smart to parallelize with slurm (outside of the programming language) if not too costly so that slurm can schedule smaller (in core-hour) jobs and get them actually run more quickly.

Basic / commonly used commands

Because slurm decides when your code will run, it is important to know what resources are being used when you’re about to run a job. Similar to the department server, conscientious usage of the servers with slurm is the best approach (this varies across what type of access you have at PSC). Additionally, given that slurm can constrain the number of core hours you use, it’s useful to know how to cancel your jobs if something is not processing correctly.

Commmand	Description	Unix similarity
squeue	get a complete list of all jobs running	top
squeue -u username	all jobs running for just a single user	top -u username
sinfo	provide information about the state of each node (look for idle nodes)
scancel	cancel jobs	slightly similar to canceling a job with a terminal multiplexer
scancel -u username	cancel all jobs for a single user (only can do with yourself)
scancel _JobID_	cancel job with id: _JobID__
sbatch _bash_script_	submit batch request with a .sh bash script	slightly similar to running a job in a terminal multiplexer

Additional documentation: slurm.schedmd, osc.edu, local astro-PSC docs: vera-doc.psc.edu

Some additional useful tools

Based on feedback we wanted to give you some additional references to getting useful information about your jobs:

1. How to see the number of jobs you have Running, Pending, Completed, and Failed: slackoverflow
2. How to understand the amount of core hours you'ved used:
   To see the time used for each individual job:

   sacct -S2020-08-09 -u username -ojobid,start,end,alloccpu,cputime | column -t

   where:
   - ojobid: job ID
   - alloccpu: number of cpus allocated
   - cputime: amount of cpu time used
   
   To get the total number of core-hours used (summing up column 5) run:

   sacct -S2020-08-09 -u username -ojobid,start,end,alloccpu,cputime | column -t | awk '{print $5}' | awk '{s+=$1}END{print s}'

3. How to transfer files: see Linux on the server commands (ssh, sftp, GUI based transferring)

 

Example of how execute jobs

To actually execute a job it’s common practice to describe what you’d like to do in a bash file. This helps keep all the specifications about the job request in one place.

An example bash script batch-script.sh for running a set of .R files (with a few inputs) can be seen below:

#!/bin/bash
#SBATCH -t 01:00:00
#SBATCH -N 1
#SBATCH --ntasks-per-node 1
#SBATCH --array=11-101:2
#SBATCH --mail-user=andrew_id@andrew.cmu.edu
#SBATCH --mail-type=END

set -x

Rscript code.R $SLURM_ARRAY_TASK_ID 100

To run this file we’d run (on the server):

sbatch batch-script.sh

The following structures should be observed:

1. #!/bin/bash [Required], this reminds the compiler that this is a bash script
2. SBATCH -t 01:00:00 [Required], this tells slurm that we expect a maximum of 1 hour to be need to run each job (if the job is running past 1 hour it will cancel - so be conservative in this estimate!)
3. SBATCH -N 1 [Required], this tells slurm that we only need 1 nodes for each job (a node contains multiple cores)
4. SBATCH --ntasks-per-node 1, this tells slurm we only need 1 core for each job (in this example we are parallelizing with slum)
5. SBATCH --array=11-101:2 this tells slurm that we will run jobs with index 11, 13, …, to 101 (the power of slurm). See slurm.schedmd ) for details on job arrays and how you can use them to parallelize your code.
6. SBATCH --mail-user=andrew_id@andrew.cmu.edu, this makes slurm send you an email when all your jobs from this file is done (so you don’t have to keep logging in to see if the job failed / completed)
7. SBATCH --mail-type=END, defines when you’ll get the email (at the end of all all jobs)
8. set -x, stores console lines from job runs into a log file (good for debugging). Without a line above saying SBATCH -o log_file_$SLURM_ARRAY_TASK_ID.tx it will select the file’s name itself. 
9. Rscript code.R $SLURM_ARRAY_TASK_ID 100, the actual job we’re running. In this case we’re running an R file with input parameters associated with our job array number in line 4 and a second parameter 100. For python users, this could easily be replaced with ipython code.py $SLURM_ARRAY_TASK_ID 100.

SLURM Demo

For a hands on demonstration of using SLURM, click here.