Bash files

Bash and .sh files

This article is still being developed - released earlier to align with Department seminar.

Bash files (often denoted with a post-script .sh to the file name) can provide a way to store multiple bash commands that can then be run with a single command:

bash my_bash_script.sh

Some common uses include:

1. looping through multiple parameters
2. creating a set of files / folders
3. edit files in minor ways

bash for-loops

Terminal level automatization can occur when you are able to view your work as a set of command-line calls, and often this can entail multiple runs of similar commands - which makes them naturally benefit from using a for-loop. Below is a very simple example of a for-loop that runs a python script 10 times (it may be useful to read Python on the server). If we suppose we put the below code in a file called my_bash_script_py.sh then we could run it with bash my_bash_script_py.sh.

#!/bin/bash
for IDX in {1..10}
  do
    echo "Step $IDX/10"
    ipython -i my_file.py $IDX
  done

Let’s break down the above code:

1. First, the #!/bin/bash is a comment line with a “bang” (aka #!) that tells the computer that you will actually be using the bash language when you evaluate the file.
2. This for-loop varies the IDX variable for integers between 1 to 10 (inclusive). To use the IDX value later we use the notation $IDX which can be seen in lines 4 and 5 (notice how it can be included inside quotes for the echo call and outside quotes for the ipython call). The 5th line takes in $IDX as the first parameter for the python file my_file.py (see Python on the server and R on the server for more details about passing arguments in from the command line).

Defining more complex bash for-loops

The above for loop is very simple and constrained about what values you are going through. The below table shows some examples of more complex calls

code	description
for VALUE in 100 200 400 ...	$VALUE takes the value 100, 200, and 400
for VARIABLE in text1 text3 text7 ...	$VARIABLE takes the text1, text3, text7, ...
$idxtakes the values 0, 2, 4, 6, 8, 10 |
EXTERNAL_NUM=10 for i in $seq(1 2 $EXTERNAL_NUM) ...	itakes odd values between 1 and$EXTERNAL_NUM(inclusive). Where we defined theEXTERNAL_NUM right before

For more examples of for-loops in bash see cyberciti.biz

Examples

Below we provide two examples (from Ben LeRoy and Mike Stanley respectively). These try to capture the complexities you can do with bash scripts.

Example 1: Parallelization

Warning: below shows how to parallelize inside a bash script. This can be dangerous since it sends processes to the background so it’s harder/impossible to cancel. This can be very bad as your processes can use too much RAM and crash your computer…

Script’s goal: to process 60 jobs (with parameters $BIDX and $INDEX) with some parallelization (maximum 6 jobs run at once) on a local computer.

#!/bin/bash

N=6 # maximum number of jobs run at once

for BIDX in {1..6}
  do
    echo "--"
    echo "Number of Sims Idx: $BIDX"
    echo "--"
    for INDEX in {1..10}
      do
        Rscript 2-assess_sensitivity.R $BIDX $INDEX default default &
        if [[ $(jobs -r -p | wc -l) -ge $N ]]; then
          wait
        fi
    done
    wait
  done
echo "all completed"

Code breakdown:

1. [Line 3] We set N=6 to capture a global variable ($N) is the total number of jobs that can be used by this script. This constraint with be identified below. Since this relates to the number of jobs run at the same time, one should be aware about the number of cores your computer has and the amount of RAM you use for each job (the later we’ll cause memory overload if you’re not careful).
2. [Line 5:9] Our first for-loop goes across $BIDX integers between 1 and 6 (making some printouts as it does so).
3. [Line 10:11] We have an inner for-loop with $INDEX taking integer values between 1 and 10.
4. [Line 12] For each of the inner for-loops, we run an .R script that takes in 4 parameters ($BIDX, $INDEX, default and default – see R_on_the_server for more details). The & at the end of the line sends the code to be run in the background.
5. [Line 13:15] Inside the inner for-loop and after the R code is run, we have an if statement (with the ending of the if statement denoted fi). The logic of the if statement (inside the [[ ]]) is started with the $-sign, and runs the command jobs -r -p (which creates a list of the jobs running in the background) and then pipes that into wc -l (which counts) the total number of these jobs running. This value is then compared (-ge $N) with N, and if greater or equal to $N, then the if statement is executed.
6. [Line 14] If the if statement is executed then it executes the function wait which just waits until the jobs sent to the background (with the &) are finished before proceeding.
7. [Line 16] The done statement completes the inner for-loop
8. [Line 17] Another wait call makes sure that all of those jobs sent to the background for the inner for-loop are done before proceeding
9. [Line 18] The done statement completes the outer for-loop
10. [Line 19] Prints out the message “all complete”. Given the second wait on line 17, this will occur only after all BIDX values are completely processed.

More information on parallelizing with bash can be found here: cyberciti.

Example 2: complex file creation, rewriting and changing preferences

The following code is complex so we will be breaking it into 4 parts (relative to the 4 for-loops it contains.) Please image all these steps were in a single bash file. Below is the header that was in the original file:

#!/bin/bash
#
# Script to set up a multijob directory
#
# 1. Creates one cmdfile for every 4 jobs (list of file paths to the element bash scripts)
# 2. Creates each bash script in the above file
# 3. Creates directory for each element in the above
# 4. Creates PBS script for each cmdfile
#
# INPUTS
# 1. Target directory where files and directories can be made
# 2. Template element bash script
# 3. Pre-compiled GEOS-Chem directory
#
# NOTE:
# - the sample element bash script is expected to be in the same directory
#   as this script
#
# STEPS FOR USE
# 1. set directory where multijob will be executed (BASE_DIR)
# 2. set directory where compiled code lives (COMPILED_CODE_DIR)
# 3. define number of ensemble elements (NUM_ELEMENTS)
#
# Author        : Mike Stanley
# Created       : October 27, 2020
# Last Modified : November 24, 2020
#==============================================================================

Step 1

Step’s goal: The goal of the code in this step is to (1) first define a set of global variables (some focused on the overall location of files) and then (2) create a set of files in a set of subfolders

# operational params
BASE_DIR=/glade/work/mcstanley/monte_carlo/multijob
COMPILED_CODE_DIR=/glade/u/home/mcstanley/gc_adj_runs/monte_carlo_element_const_start
NUM_ELEMENTS=20

# stems
SCRIPT_STEM=element
DIR_STEP=ens
LAUNCH_STEM=launch_file

# create and fill the cmdfile
CMDFILE_NUM=0
FILE_NUM=0
for i in $(seq 0 $( expr ${NUM_ELEMENTS} - 1 ))
do
    echo $BASE_DIR/${SCRIPT_STEM}_${i}.sh >> $BASE_DIR/cmdfile_$FILE_NUM

    # update indices
    CMDFILE_NUM=$((CMDFILE_NUM + 1))

    if [ $((CMDFILE_NUM % 4)) -eq 0 ]; then
        FILE_NUM=$((FILE_NUM + 1))
    fi

done

Code breakdown:

1. [Line 2:4, 7:9] define global variables
2. [Line 12:13] define global variables for the for-loop. The first is just an index of the for loop (@Mike - why not just use $i?). The second will be updated every 4 files are created and helps define new folders.
3. [Line 14:15] starts a for-loop that will see $i go from 0 to 19 (inclusively), given that on line 4 we defined $NUM_ELEMENTS=20.
4. [Line 16] The function call echo _1_ >> _2_ adds the _1_ as the final text line of the file _2_ (it also creates _2_ if it doesn’t already exist). _1_ is the pasting of the $BASE_DIR, /, $SCRIPT_STEM, _, $i and .sh (as similar thing is true for _2_.
5. [Line 18] (@Mike - again I’m not sure why this included). This line just updates the $CMDFILE_NUM to 1 plus it’s original value
6. [Line 20:22] This is an if statement that checks if the variable $CMDFILE_NUM’s remainder after defining by 4 (mod = %) equals (-eq) 0. If so it then updates the $FILE_NUM to 1 plus it’s original value. Ends the if statement with fi
7. [Line 24] ends the for loop from line 12

Step 2

Step’s goal: create additional bash files (associated with numbering above) and edit those bash files to have ENS_NUM also associated with the new name of the file.

# copy sample bash script and alter element number
for i in $(seq 0 $( expr ${NUM_ELEMENTS} - 1 ))
do
    # create the copy for element i
    cp ./sample_element.sh $BASE_DIR/${SCRIPT_STEM}_${i}.sh

    # alter the ENS_NUM= line
    # NOTE: there might be platform dependence with the sed command
    replace_str="ENS_NUM=${i}"
    sed -i.bu "s/ENS_NUM=0/${replace_str}/" $BASE_DIR/${SCRIPT_STEM}_${i}.sh

    # remove the created backup
    rm $BASE_DIR/${SCRIPT_STEM}_${i}.sh.bu

done

Code breakdown:

1. [Line 2:3, 15] see lines 14:15 and 24 in step 1.
2. [Line 5] make a copy of ./sample_element.sh with name and in the directory $BASE_DIR with name $SCRIPT_STEM_${i}.sh (so for the first for-loop step, we’d have it be called element_0.sh (remember $BASE_DIR was defined in step 1’s line 2).
3. [Line 9] define a new variable replacement_string
4. [Line 10] use the sed command to relace all strings of the form ENS_NUM=0 with your $replace_str in the newly created fill (from line 5). See geeksforgeeks for more details on the power of sed.
5. [Line 11] Since we have to extra parameter sed -i.bu in line 10, we delete a back-up file created from sed (@Mike - I’m not 100% sure this is true - and is it needed?)

Step 3

Step’s goal: change file privileges of script files and create and fill new directories associated with new scripts (from step 2)

# give executable privileges to the above scripts
chmod ugo+x $BASE_DIR/${SCRIPT_STEM}_*

# create the code directories for each element
for i in $(seq 0 $( expr ${NUM_ELEMENTS} - 1 ))
do
    # make the directory
    mkdir $BASE_DIR/ens_${i}

    # make the directory by copying over contents
    cp -r $COMPILED_CODE_DIR/* $BASE_DIR/ens_${i}

done

Code breakdown:

1. [Line 2] change file privileges for all the new script files using chmod. Note that chmod ugo+x updates the executability privileges of the file to the user that owns it, members in the user’s group and those outside the user’s group. See computerhope for more references. This was necessary due to default file privileges on the server and how user structure was setup.
2. [Line 5:6] start of a for-loop for variable $i ranging between 0 and $NUM_ELMENTS - 1 - aka for all newly make files in Step 2.
3. [Line 8] make new directories assocated with each new script (@Mike: should this be mkdir ${BASE_DIR}/${DIR_STEP}_${i}? )
4. [Line 11] copy the full directory (-r: recursively) of the $COMPILED_CODE_DIR/ into our newly make folders

Step 4

Step’s goal: copy a launch script into a set of new folders, then update the copied script by changing the name of the cmdfile it is using. (@Mike: is the LAUNCH_STEP folders already created?)

# create PBS run files for each cmdfile
CMD_FILE_NMS=$BASE_DIR/cmdfile*

COUNT=0
for file in $CMD_FILE_NMS
do
    # create the copy for cmdfile i
    cp ./sample_launch_script $BASE_DIR/${LAUNCH_STEM}_${COUNT}

    # replace the cmdfile name
    replace_str="cmdfile_${COUNT}"
    sed -i.bu "s/\/cmdfile/\/${replace_str}/" $BASE_DIR/${LAUNCH_STEM}_${COUNT}

    # remove the created backup
    rm $BASE_DIR/${LAUNCH_STEM}_${COUNT}.bu

    COUNT=$((COUNT + 1))
done

Code breakdown:

1. [Line 2] define a list of variable names starting with a certain structure
2. [Line 4, 17] define a variable that starts at 0 and add 1 to it each time the following loop runs
3. [Line 5:6, 14] a for-loop across the file names in the list defined in line 2.
4. [Line 8] copy file ./sample_launch_script into file into new older
5. [Line 10:11, 13] replace the string /cmdfile to /cmdfile_[i] where i in a new index (in the file you just created with the copy in line 8). Line 13 removes temporary file creates with the replacement