Ajshen at 02:52, 29 June 2026

2026-06-29T02:52:17Z

← Older revision		Revision as of 02:52, 29 June 2026
Line 1:		Line 1:
			[[Category:Terminal and Command Line]]

	== Bash and <code>.sh</code> files ==		== Bash and <code>.sh</code> files ==

Skipper: Created page with "== Bash and `.sh` files == '''This article is still being developed - released earlier to align with Department seminar.''' Bash files (often denoted with a post-..."

2024-03-30T00:01:49Z

Created page with "== Bash and <code>.sh</code> files == '''This article is still being developed - released earlier to align with Department seminar.''' Bash files (often denoted with a post-..."

New page

== Bash and <code>.sh</code> files ==

'''This article is still being developed - released earlier to align with Department seminar.'''

Bash files (often denoted with a post-script <code>.sh</code> to the file name) can provide a way to store multiple bash commands that can then be run with a single command:

<syntaxhighlight lang="bash">bash my_bash_script.sh</syntaxhighlight>
Some common uses include:

# looping through multiple parameters
# creating a set of files / folders
# edit files in minor ways

=== <code>bash</code> for-loops ===

Terminal level automatization can occur when you are able to view your work as a set of command-line calls, and often this can entail multiple runs of similar commands - which makes them naturally benefit from using a for-loop. Below is a very simple example of a for-loop that runs a <code>python</code> script 10 times (it may be useful to read [[Python_on_the_server|Python on the server]]). If we suppose we put the below code in a file called <code>my_bash_script_py.sh</code> then we could run it with <code>bash my_bash_script_py.sh</code>.

<syntaxhighlight lang="bash">#!/bin/bash
for IDX in {1..10}
do
echo "Step $IDX/10"
ipython -i my_file.py $IDX
done</syntaxhighlight>
Let’s break down the above code:

# First, the <code>#!/bin/bash</code> is a comment line with a “bang” (aka <code>#!</code>) that tells the computer that you will actually be using the <code>bash</code> language when you evaluate the file.
# This for-loop varies the <code>IDX</code> variable for integers between 1 to 10 (inclusive). To use the <code>IDX</code> value later we use the notation <code>$IDX</code> which can be seen in lines 4 and 5 (notice how it can be included inside quotes for the <code>echo</code> call and outside quotes for the <code>ipython</code> call). The 5th line takes in <code>$IDX</code> as the first parameter for the python file <code>my_file.py</code> (see [[Python_on_the_server|Python on the server]] and [[R_on_the_server|R on the server ]] for more details about passing arguments in from the command line).

==== Defining more complex <code>bash</code> for-loops ====

The above for loop is very simple and constrained about what values you are going through. The below table shows some examples of more complex calls

{| class="wikitable"
!width="31%"| code
!width="68%"| description
|-
| <code>for VALUE in 100 200 400 ...</code>
| <code>$VALUE</code> takes the value <code>100</code>, <code>200</code>, and <code>400</code>
|-
| <code>for VARIABLE in <code>text1 text3 text7 ...</code>
| <code>$VARIABLE</code> takes the <code>text1</code>, <code>text3</code>, <code>text7</code>, <code>...</code>
|-
| <code>for idx in {0..10..2} ...</code> |<code>$idx</code>takes the values <code>0</code>, <code>2</code>, <code>4</code>, <code>6</code>, <code>8</code>, <code>10</code> |
|-
|<syntaxhighlight lang="bash">EXTERNAL_NUM=10
for i in $seq(1 2 $EXTERNAL_NUM) ...</syntaxhighlight>
| <code>i</code>takes odd values between 1 and<code>$EXTERNAL_NUM</code>(inclusive). Where we defined the<code>EXTERNAL_NUM</code> right before
|-
|}

For more examples of for-loops in bash see [https://www.cyberciti.biz/faq/bash-for-loop/ cyberciti.biz]

== Examples ==

Below we provide two examples (from Ben LeRoy and Mike Stanley respectively). These try to capture the complexities you can do with bash scripts.

=== Example 1: Parallelization ===

''Warning: below shows how to parallelize inside a bash script. This can be dangerous since it sends processes to the background so it’s harder/impossible to cancel. This can be very bad as your processes can use too much RAM and crash your computer…''

'''Script’s goal:''' to process 60 jobs (with parameters <code>$BIDX</code> and <code>$INDEX</code>) with some parallelization (maximum 6 jobs run at once) on a local computer.

<syntaxhighlight lang="bash">#!/bin/bash

N=6 # maximum number of jobs run at once

for BIDX in {1..6}
do
echo "--"
echo "Number of Sims Idx: $BIDX"
echo "--"
for INDEX in {1..10}
do
Rscript 2-assess_sensitivity.R $BIDX $INDEX default default &
if [[ $(jobs -r -p | wc -l) -ge $N ]]; then
wait
fi
done
wait
done
echo "all completed"
</syntaxhighlight>
'''Code breakdown:'''

# [Line 3] We set <code>N=6</code> to capture a global variable (<code>$N</code>) is the total number of jobs that can be used by this script. This constraint with be identified below. Since this relates to the number of jobs run at the same time, one should be aware about the number of cores your computer has and the amount of RAM you use for each job (the later we’ll cause memory overload if you’re not careful).
# [Line 5:9] Our first for-loop goes across <code>$BIDX</code> integers between 1 and 6 (making some printouts as it does so).
# [Line 10:11] We have an inner for-loop with <code>$INDEX</code> taking integer values between 1 and 10.
# [Line 12] For each of the inner for-loops, we run an <code>.R</code> script that takes in 4 parameters (<code>$BIDX</code>, <code>$INDEX</code>, <code>default</code> and <code>default</code> – see [[R_on_the_server]] for more details). The <code>&</code> at the end of the line sends the code to be run in the background.
# [Line 13:15] Inside the inner for-loop and after the <code>R</code> code is run, we have an <code>if</code> statement (with the ending of the if statement denoted <code>fi</code>). The logic of the <code>if</code> statement (inside the <code>[[ ]]</code>) is started with the <code>$</code>-sign, and runs the command <code>jobs -r -p</code> (which creates a list of the jobs running in the background) and then pipes that into <code>wc -l</code> (which counts) the total number of these jobs running. This value is then compared (<code>-ge $N</code>) with N, and if greater or equal to <code>$N</code>, then the if statement is executed.
# [Line 14] If the <code>if</code> statement is executed then it executes the function <code>wait</code> which just waits until the jobs sent to the background (with the <code>&</code>) are finished before proceeding.
# [Line 16] The <code>done</code> statement completes the inner for-loop
# [Line 17] Another <code>wait</code> call makes sure that all of those jobs sent to the background for the inner for-loop are done before proceeding
# [Line 18] The <code>done</code> statement completes the outer for-loop
# [Line 19] Prints out the message “all complete”. Given the second <code>wait</code> on line 17, this will occur only after all <code>BIDX</code> values are completely processed.

More information on parallelizing with bash can be found here: [https://www.cyberciti.biz/faq/how-to-run-command-or-code-in-parallel-in-bash-shell-under-linux-or-unix/ cyberciti].

=== Example 2: complex file creation, rewriting and changing preferences ===

The following code is complex so we will be breaking it into 4 parts (relative to the 4 for-loops it contains.) Please image all these steps were in a single bash file. Below is the header that was in the original file:

<syntaxhighlight lang="bash">#!/bin/bash
#
# Script to set up a multijob directory
#
# 1. Creates one cmdfile for every 4 jobs (list of file paths to the element bash scripts)
# 2. Creates each bash script in the above file
# 3. Creates directory for each element in the above
# 4. Creates PBS script for each cmdfile
#
# INPUTS
# 1. Target directory where files and directories can be made
# 2. Template element bash script
# 3. Pre-compiled GEOS-Chem directory
#
# NOTE:
# - the sample element bash script is expected to be in the same directory
# as this script
#
# STEPS FOR USE
# 1. set directory where multijob will be executed (BASE_DIR)
# 2. set directory where compiled code lives (COMPILED_CODE_DIR)
# 3. define number of ensemble elements (NUM_ELEMENTS)
#
# Author : Mike Stanley
# Created : October 27, 2020
# Last Modified : November 24, 2020
#==============================================================================
</syntaxhighlight>

==== Step 1 ====

'''Step’s goal:''' The goal of the code in this step is to (1) first define a set of global variables (some focused on the overall location of files) and then (2) create a set of files in a set of subfolders

<syntaxhighlight lang="bash"># operational params
BASE_DIR=/glade/work/mcstanley/monte_carlo/multijob
COMPILED_CODE_DIR=/glade/u/home/mcstanley/gc_adj_runs/monte_carlo_element_const_start
NUM_ELEMENTS=20

# stems
SCRIPT_STEM=element
DIR_STEP=ens
LAUNCH_STEM=launch_file

# create and fill the cmdfile
CMDFILE_NUM=0
FILE_NUM=0
for i in $(seq 0 $( expr ${NUM_ELEMENTS} - 1 ))
do
echo $BASE_DIR/${SCRIPT_STEM}_${i}.sh >> $BASE_DIR/cmdfile_$FILE_NUM

# update indices
CMDFILE_NUM=$((CMDFILE_NUM + 1))

if [ $((CMDFILE_NUM % 4)) -eq 0 ]; then
FILE_NUM=$((FILE_NUM + 1))
fi

done</syntaxhighlight>
'''Code breakdown:'''

# [Line 2:4, 7:9] define global variables
# [Line 12:13] define global variables for the for-loop. The first is just an index of the for loop (@Mike - why not just use <code>$i</code>?). The second will be updated every 4 files are created and helps define new folders.
# [Line 14:15] starts a for-loop that will see <code>$i</code> go from 0 to 19 (inclusively), given that on line 4 we defined <code>$NUM_ELEMENTS=20</code>.
# [Line 16] The function call <code>echo _1_ >> _2_</code> adds the <code>_1_</code> as the final text line of the file <code>_2_</code> (it also creates <code>_2_</code> if it doesn’t already exist). <code>_1_</code> is the pasting of the <code>$BASE_DIR</code>, <code>/</code>, <code>$SCRIPT_STEM</code>, <code>_</code>, <code>$i</code> and <code>.sh</code> (as similar thing is true for <code>_2_</code>.
# [Line 18] (@Mike - again I’m not sure why this included). This line just updates the <code>$CMDFILE_NUM</code> to <code>1</code> plus it’s original value
# [Line 20:22] This is an <code>if</code> statement that checks if the variable <code>$CMDFILE_NUM</code>’s remainder after defining by 4 (<code>mod</code> = <code>%</code>) equals (<code>-eq</code>) <code>0</code>. If so it then updates the <code>$FILE_NUM</code> to <code>1</code> plus it’s original value. Ends the <code>if</code> statement with <code>fi</code>
# [Line 24] ends the for loop from line 12

==== Step 2 ====

'''Step’s goal:''' create additional bash files (associated with numbering above) and edit those bash files to have <code>ENS_NUM</code> also associated with the new name of the file.

<syntaxhighlight lang="bash"># copy sample bash script and alter element number
for i in $(seq 0 $( expr ${NUM_ELEMENTS} - 1 ))
do
# create the copy for element i
cp ./sample_element.sh $BASE_DIR/${SCRIPT_STEM}_${i}.sh

# alter the ENS_NUM= line
# NOTE: there might be platform dependence with the sed command
replace_str="ENS_NUM=${i}"
sed -i.bu "s/ENS_NUM=0/${replace_str}/" $BASE_DIR/${SCRIPT_STEM}_${i}.sh

# remove the created backup
rm $BASE_DIR/${SCRIPT_STEM}_${i}.sh.bu

done</syntaxhighlight>
'''Code breakdown:'''
# [Line 2:3, 15] see lines 14:15 and 24 in step 1.
# [Line 5] make a copy of <code>./sample_element.sh</code> with name and in the directory <code>$BASE_DIR</code> with name <code>$SCRIPT_STEM_${i}.sh</code> (so for the first for-loop step, we’d have it be called <code>element_0.sh</code> (remember <code>$BASE_DIR</code> was defined in step 1’s line 2).
# [Line 9] define a new variable <code>replacement_string</code>
# [Line 10] use the <code>sed</code> command to relace all strings of the form <code>ENS_NUM=0</code> with your <code>$replace_str</code> in the newly created fill (from line 5). See [https://www.geeksforgeeks.org/sed-command-in-linux-unix-with-examples/ geeksforgeeks] for more details on the power of <code>sed</code>.
# [Line 11] Since we have to extra parameter <code>sed -i.bu</code> in line 10, we delete a back-up file created from <code>sed</code> (@Mike - I’m not 100% sure this is true - and is it needed?)

==== Step 3 ====

'''Step’s goal:''' change file privileges of script files and create and fill new directories associated with new scripts (from step 2)

<syntaxhighlight lang="bash"># give executable privileges to the above scripts
chmod ugo+x $BASE_DIR/${SCRIPT_STEM}_*

# create the code directories for each element
for i in $(seq 0 $( expr ${NUM_ELEMENTS} - 1 ))
do
# make the directory
mkdir $BASE_DIR/ens_${i}

# make the directory by copying over contents
cp -r $COMPILED_CODE_DIR/* $BASE_DIR/ens_${i}

done</syntaxhighlight>
'''Code breakdown:'''
#[Line 2] change file privileges for all the new script files using <code>chmod</code>. Note that <code>chmod ugo+x</code> updates the e<code>x</code>ecutability privileges of the file to the <code>u</code>ser that owns it, members in the user’s <code>g</code>roup and those <code>o</code>utside the user’s group. See [https://www.computerhope.com/unix/uchmod.htm computerhope] for more references. This was necessary due to default file privileges on the server and how user structure was setup.
# [Line 5:6] start of a for-loop for variable <code>$i</code> ranging between <code>0</code> and <code>$NUM_ELMENTS - 1</code> - aka for all newly make files in Step 2.
# [Line 8] make new directories assocated with each new script (@Mike: should this be <code>mkdir ${BASE_DIR}/${DIR_STEP}_${i}</code>? )
# [Line 11] copy the full directory (<code>-r</code>: recursively) of the <code>$COMPILED_CODE_DIR/</code> into our newly make folders

==== Step 4 ====

'''Step’s goal:''' copy a launch script into a set of new folders, then update the copied script by changing the name of the cmdfile it is using. (@Mike: is the LAUNCH_STEP folders already created?)

<syntaxhighlight lang="bash"># create PBS run files for each cmdfile
CMD_FILE_NMS=$BASE_DIR/cmdfile*

COUNT=0
for file in $CMD_FILE_NMS
do
# create the copy for cmdfile i
cp ./sample_launch_script $BASE_DIR/${LAUNCH_STEM}_${COUNT}

# replace the cmdfile name
replace_str="cmdfile_${COUNT}"
sed -i.bu "s/\/cmdfile/\/${replace_str}/" $BASE_DIR/${LAUNCH_STEM}_${COUNT}

# remove the created backup
rm $BASE_DIR/${LAUNCH_STEM}_${COUNT}.bu

COUNT=$((COUNT + 1))
done</syntaxhighlight>
'''Code breakdown:'''
# [Line 2] define a list of variable names starting with a certain structure
# [Line 4, 17] define a variable that starts at <code>0</code> and add <code>1</code> to it each time the following loop runs
# [Line 5:6, 14] a for-loop across the file names in the list defined in line 2.
# [Line 8] copy file <code>./sample_launch_script</code> into file into new older
# [Line 10:11, 13] replace the string <code>/cmdfile</code> to <code>/cmdfile_[i]</code> where i in a new index (in the file you just created with the copy in line 8). Line 13 removes temporary file creates with the replacement

Bash files - Revision history

Ajshen at 02:52, 29 June 2026

Skipper: Created page with "== Bash and .sh files == '''This article is still being developed - released earlier to align with Department seminar.''' Bash files (often denoted with a post-..."

Skipper: Created page with "== Bash and `.sh` files == '''This article is still being developed - released earlier to align with Department seminar.''' Bash files (often denoted with a post-..."