View source for Running Docker locally and on Hydras


[[Category:Containers and Virtual Environments]]

= Introduction =

This document explains the functionalities of the package created by Lucas Kania to run algorithms locally and remotely inside Docker, which is available for download at this [https://github.com/cmustatistics/docker link].

The focus is on explaining the salient points of different scripts so that you can modify them to suit your algorithm.&nbsp;We won't explain the technicalities involved in using <code>Docker</code>, <code>tmux</code>&nbsp;and <code>ssh</code>. For a basic introduction to those tools, the reader is referred to the relevan links in the [https://www.stat.cmu.edu/computewiki/index.php wiki]. The example project is a simplified version of “[https://doi.org/10.1007/978-3-030-86523-8_32 Sparse&nbsp;Information Filter for Fast Gaussian Process Regression]” by Kania et al. (2021), which proposes a method for training sparse Gaussian processes via mini-batches rather than full-batches.&nbsp;

The structure of the repository&nbsp;is as follows:

*''Dockerfile''&nbsp;configures a docker image. At a basic level, it is a declaration of the environment where the algorithm will run. 
*''DockerfileGPU''&nbsp;configures a docker image with GPU support. 
*''.dockerignore''&nbsp;configures exceptions when creating the docker image. 
*''rundocker.sh''&nbsp;runs the algorithm inside the container. 
*''src''&nbsp;contains the source code of the algorithm. 
*''data'' contains the data used for this usage example. 
*''results'' is the folder where the results of the example&nbsp;are saved. 
*''utilities''&nbsp;contains useful scripts for running algorithms remotely. 
*''paper.pdf''&nbsp;is the paper whose source code is used for this example. The algorithms implemented inside the ''src&nbsp;''folder are explained in it. ''supp.pdf'' contains additional information. 

Many of the code snippets shown in this document have been substantially simplified to reflect the essence of the scripts. However, the scripts in the root folder differ since they target a specific algorithm rather than a generic one.&nbsp;

= Running docker locally in your computer =

== Install docker ==

Get the latest version of docker desktop from this [https://docs.docker.com/desktop/ link]. Once the installation finishes, please start docker desktop.

== Configure a docker image ==

''Dockerfile''&nbsp;configures the docker image, that is, the environment containing all the dependencies necessary for running your algorithm.&nbsp;

We start by declaring a base image on which we will install further dependencies. For example, the following code provides a ''linux''&nbsp;system with ''python''&nbsp;3.8 support.&nbsp;
<pre>FROM python:3.8-slim-buster
</pre>

Consult the ''Dockerfile''&nbsp;for further instructions on how to use a base image with ''Pytorch''&nbsp;and ''Tensorflow'' with GPU support.

On it, we might install further dependencies via ''pip''. It is recommended to specify the version of the required dependencies. Otherwise, ''pip'' will install the latest version compatible with our other packages making the docker image non-deterministic.&nbsp;
<pre># We install additional packages required for this particular project
RUN pip install pandas==2.0.3</pre>

Then, we copy the code required to run the algorithm and experiments&nbsp;
<pre># We copy the folder containing the source code of our algorithm into the folder /program/src in the container
COPY src /program/src
</pre>

We do not copy the data or results folders into the container. They will be mounted to the Docker container during runtime.

== Building a docker container ==

Our next step is to start a Docker container based on our image. The script ''rundocker.sh''&nbsp;simplifies the building procedure. To wit, it&nbsp;

*creates a docker container based on the image defined in ''Dockerfile'' 
*mounts the data folder instead of copying it into the container. This avoids making images unnecessarily large due to large datasets.&nbsp; 
*mounts the results folder instead of copying it into the container. Therefore, if the computer crashes, the saved data will remain after reboot. 
*runs an algorithm inside the container. 

The most important part of the script is indicating the location of your algorithm's executable. Everything else in the file might be left untouched if you follow the conventions adopted by this tutorial.
<pre>#####################################################
# Location of the executable inside the container
#####################################################
readonly executable=/program/experiments/runner.py
</pre>

== Running an algorithm in the docker container ==

To run your container, execute the script ''rundocker.sh''. You will immediately see the output of our algorithm.&nbsp;
<pre>bash rundocker.sh
</pre>

If you wish to pass arguments to your executable, you can do so by running
<pre>bash rundocker.sh your_arguments    </pre>

Then, you can parse <code>your_arguments</code> inside your script.

= Remotely running one instance of the algorithm =

== Configure the server ==

A ssh connection to the server can be configured at <code>~/.ssh/config</code>. For example, using the following code, you can configure access to <code>hydra1.stat.cmu.edu</code>.
<pre>Host hydra1
    Hostname hydra1.stat.cmu.edu
    Port 22
    user your_usename
    IdentityFile ~/.ssh/your_public_key_file
    ServerAliveInterval 180
</pre>

Thus, we execute&nbsp;<code>ssh hydra1</code> in the terminal, ''ssh''&nbsp;will automatically connect to <code>hydra1.stat.cmu.edu</code>&nbsp;using the public key <code>~/.ssh/your_public_key_file</code>.&nbsp;

== Copying your files to the remote server ==

We can copy all the files in this folder to ''hydra1''&nbsp;using <code>scp</code>:&nbsp;<code>scp -r $PWD hydra1:~/remote/sparsegp</code>

== Remote execution and monitoring ==

After running the command in the previous section, ''rundocker.sh''&nbsp;will be located at <code>~/remote/rundocker.sh</code>. You can use the utility ''distribute.sh''&nbsp;to connect to the remote server and run your ''rundocker.sh''&nbsp;script.
<pre>bash distribute.sh --server hydra1 --session TEST --rundocker "~/remote/sparsegp"
</pre>

''distribute.sh''&nbsp;connects to the ''hydra1''&nbsp;server, and runs the ''rundocker.sh''&nbsp;script inside a <code>tmux</code>&nbsp;session called TEST.&nbsp;To observe the output of our algorithm, we can use the ''process.sh''&nbsp;utility. Specify the <code>tmux</code>&nbsp;session name and the server as follows.&nbsp;
<pre>bash process.sh --session TEST --server hydra1
</pre>

= Remotely running multiple instances of your algorithm =

You can run one instance per server of your algorithm with different parameters in each server. This is useful if you run the same experiments multiple times with different parameters.

== Configure the servers ==

We configure four servers at <code>~/.ssh/config</code>, two ''ghidorahs''&nbsp;and two ''hydras''.
<pre>Host ghidorah2
    Hostname ghidorah2.stat.cmu.edu
    Port 22
    user your_usename
    IdentityFile ~/.ssh/your_public_key_file
    ServerAliveInterval 180
Host ghidorah3
    Hostname ghidorah3.stat.cmu.edu
    Port 22
    user your_usename
    IdentityFile ~/.ssh/your_public_key_file
    ServerAliveInterval 180
Host hydra1
    Hostname hydra1.stat.cmu.edu
    Port 22
    user your_usename
    IdentityFile ~/.ssh/your_public_key_file
    ServerAliveInterval 180
Host hydra2
    Hostname hydra2.stat.cmu.edu
    Port 22
    user your_usename
    IdentityFile ~/.ssh/your_public_key_file
    ServerAliveInterval 180
</pre>

Subsequently, we specify the servers where you want to run the instances of your algorithm in ''servers.sh.''
<pre># Declare servers to be used for running your scripts
servers=(
 ghidorah2
 ghidorah3
 hydra1
 hydra2
)
</pre>

== Remote execution and monitoring ==

The process is analogous to running one instance, but we must iterate over the declared servers in ''server.sh''. The utility ''experiments.sh''&nbsp;gives one example of how to achieve this. In short, we load the server configuration by executing&nbsp;
<pre>#############################################
# load servers
#############################################
. servers.sh
</pre>

Then, we declare an array with the parameters that we want to pass to the algorithm:
<pre>#############################################
# Declare parameters that change from server to server
#############################################
parameters=(
 param_for_ghidorah2
 param_for_ghidorah3
 param_for_hydra1
 param_for_hydra2
)
</pre>

and iterate over the servers:
<pre>#############################################
# We iterate over the servers and run the script
# with different parameters in each one of them
#############################################
for i in "${!parameters[@]}"; do
    ./distribute.sh --server "${servers[i]}"      
    --session PARAMETERS                          
    --rundocker "~/remote/"                     
    "${parameters[i]}"                            
done
</pre>

We can run the script via bash:&nbsp;<code>bash experiments.sh</code>

To monitor the four instances, we can execute&nbsp;<code>bash process.sh --session TEST</code>. Note that we specify the session but not the server. The utility ''process.sh''&nbsp;will iterate over the servers declared in ''servers.sh''&nbsp;and connect to each session.

== Resetting everything ==

If you want to stop the container in all the servers specified in ''servers.sh'', execute <code>bash clean.sh</code>.