Docker, and why-you-should-use-it-for-your-research
First of all, Docker is a software platform that allows you to build, test, and deploy applications quickly. Docker packages software into standardized units called containers that have everything the software needs to run including libraries, system tools, code, and runtime. Before talking more about Docker, let's start from what we probably care more about in the Stats department. That is, why do I need Docker?
Why you should use Docker in your research?
Imagine you’re trying to run a script that needs a particular version of a package. You install that version but it doesn’t seem to work. Or you already have a different version installed for another project and can’t change it. Or the version it needs is really old and isn’t available to download anymore. You spend hours installing different combinations of packages and eventually you get it working, but you’re not sure exactly what fixed it and you couldn’t repeat the same steps in the future if you wanted to exactly reproduce the environment you’re now working in. Many projects require an interconnected web of dependencies, so there are a lot of things that can go wrong when you’re trying to get everything set up. There are a few tools that can help with some of these problems. For Python you can use virtual environments with conda, for instance.
If people who download your code today can struggle to get it running, how will it be years from now when the version of that package you used isn’t around anymore and the current version is incompatible? We have a solution, containers! And specifically, Docker. (There are other options, such as Kubernetes, but we are only going to talk about Docker here.)
Using containers could seriously improve the reproducibility of your research. If you bundle up your code and data in a Docker image, and publish that image alongside your papers, anyone in the world will be able to re-run your code and get the same results with almost no effort. That includes yourself a few years from now, when you don’t remember how your code works and half of its dependencies aren’t available to install any more.
What is Docker and how it works
As already mentioned above, Docker is a software platform that allows you to build, test, and deploy applications quickly. Docker packages software into standardized units called containers that have everything the software needs to run including libraries, system tools, code, and runtime. Using Docker, you can quickly deploy and scale applications into any environment and know your code will run. Docker works by providing a standard way to run your code. Docker is an operating system for containers.
Docker Image
It is a kind of ready-to-use software read-only template crafted with source codes, libraries, external dependencies, tools, and other miscellaneous files that are needed for any software application to run successfully on any platform or OS.
Docker Container
As we discussed, docker images are nothing but a read-only template that can’t be executed by themselves and cannot run or start. If that is the limitation how can one make a real value out of it? Well, the answer lies in the concept of the Docker container, which is basically a container as we described it in another article. A container is nothing but a box that has the ability to run the docker image templates. The moment you create a container using those immutable images you essentially end up creating a read-write copy of that filesystem (docker image) inside the given container. This adds a container layer which helps you to modify the entire copy of the given Docker image.
Docker Images vs. Docker Containers
- The key difference between a Docker image Vs a container is that a Docker image is a read-only immutable template that defines how a container will be realized. A Docker container is a runtime instance of a Docker image that gets created when the
$ docker runcommand is implemented. - Before the docker container can even exist, docker templates/images are built using
$ docker buildCLI. - Docker image templates can exist in isolation but containers can't exist without images.
- So docker image is an integral part of containers that differs only because of their objectives.
- Docker images can’t be paused or started but a Docker container is a run time instance that can be started or paused.
Useful resources
Online you can find a plethora of exhaustive articles and guides that talk about Docker and its functionalities, besides explaining how to use it. It would be pointless to repeat all of that content here, so I will limit myself to redirecting you to a few useful learning resources:
Tutorials
See the next article Tutorials.
How can it fit in my Workflow?
Your development cycle will probably look something like this: First you’ll probably outline an initial version of the code, and then write a Dockerfile containing the instructions for installing the dependencies and running the code. Then it’s basically the same as what you’d normally do. As you’re working on the code, you’d iterate by building an image and then running that image as a container to test it. (With more advanced usage you can often avoid building a new image every time you run it, by mounting the working directory into the container at runtime.) Once the code is ready you can make it available by publishing the Docker image. To publish the Docker image, you can simply publish the Dockerfile along with your code an/or export the image as a tar file and upload that somewhere.
There are also a few packages which will allow you to run containers in high performance computing environments. Two popular ones are Singularity and Shifter. These will allow you to develop locally using Docker, and then convert your Docker image to run on your HPC cluster. That means the environment it runs in on the cluster will be identical to your development environment, so you won’t run into any surprises when it’s time to run it. Talk to your institution’s IT/HPC people to find out what options are available to you.
To conclude
Docker itself is open source, and all the core file formats and designs are standardised by the Open Container Initiative. Besides Docker, other OCI members include tech giants such as Amazon, Facebook, Microsoft, Google, and lots of others. The technology is designed to be future proof and it isn’t going away, and you won’t be locked into any one vendor’s products by using it. If you package your software in a Docker container you can be reasonably certain it will still run years, or decades, from now. You can install Docker for free by downloading the community edition.
