R on the server

The department's Hydra cluster has R installed on each machine. It also has RStudio Server running on each machine, so there are two ways to run R code.

Running R on the server

Command line

On the Hydras, or on other servers that do not support RStudio, log into the server via SSH first. After getting onto the server (see Linux and the department server), you can start using R but opening an interactive session on the console with

R

You’ll be able to run code (except for visualization as the server can’t display graphics), from the console.

RStudio Server

RStudio Server is running on each Hydra. Choose a Hydra with low usage. For example, suppose you choose hydra3. Then type the following URL into your web browser:

http://hydra3.stat.cmu.edu:8787/

This will open RStudio Server. Use your statistics account (the same one you use to log in via SSH) to log in.

You will be able to use the RStudio files pane to view all your files on the hydras. It looks just like RStudio on your own computer. Files are shared across machines, but your RStudio session is specific to the hydra you started it on. Return to the same hydra to view your running code or output.

Library Management

Compared to your personal computer (and your personal computer using RStudio), installing packages has a few nuances.

1. The first time installing a package, e.g.

   server$ R
   
   R > install.packages("tidyverse")

   you’ll be asked to set up a local folder to store your files.

2. The server’s version of R may different than you personal computer. It will also occasionally update. This will require you to re-install packages (seeRenv’s wiki (R package environments) for a way to make this less painful).
3. Compared to RStudio, you may need to debug package installation and dependence structure. Good luck!

Running Jobs

Beyond just opening up an interactive R session with R, you can run files in 2 ways (as described below).

Rscript

Rscript is a more commonly used tool to run .R files, as it allows you to pass additional arguments to the file. For example:

Rscript file.R arg1 arg2

In order to use arg1 and arg2 in your R code, you’ll access them (as strings) using commandArgs(), for example:

args <- commandArgs(trailingOnly = TRUE)
if (length(args) > 0) {
  first_parameter <- args[1]
  second_parameter <- as.numeric(args[2])
  ...
}

Rscript also allows for standard output (printed to the terminal while running).

Example uses of arguments

Examples of argument values abound in statistical analysis. If you're using [Slurm|Slurm] you may use arguments to parallelize a task. These types of tasks could be running through complicated data observations, examining outcomes across a grid of hyperparameters, etc. It could also be useful to pass in the name of a file to process or folder to pull from.

R CMD BATCH

R CMD BATCH is an older approach (compared to Rscript you cannot pass in additional arguments nor see printouts to the standard output, but a file_name.Rout file will be generated with all print outs of the code.

R CMD BATCH file.R

If you don't have any input arguments to access and want to save errors and messages that came from running your .R file, this might be a better approach.

Additional useful tools

When approaching running code on different servers (and even on your own computer), it may be useful to understand the amount of time an analysis (or a part of an analysis) will run and how much memory it uses. This can help you decide what computing resources you need / what would make your analysis feasible. Knowing this information can also help you decide if you need to make your code more efficient (which can be done with the help of profiles and potentially C++ functions - link to come soon).

1. Estimating time usage: To measure the amount of time code takes to run, one approach is to put

   start <- Sys.time()

   at the start and at the end

   used_time <- Sys.time() - start
   print(used_time)

2. Understanding memory usage: When you are running code in R or RStudio, you can get some understanding of how large an object is (and therefore get a lower bound on your potential RAM usage) using the package pryr. Calling pryr::object_size(object_name) will tell you what the approximate size of the object is. See adv-r on the package usage. For example,

   R> library(pryr)
   R> dm_small <- as.matrix(dist(1:10))
   R> object_size(dm_small)
   2.1 kB
   R> dm_medium <- as.matrix(dist(1:10-))
   R> object_size(dm_medium)
   87.7 kB
   R> dm_large <- as.matrix(dist(1:1000))
   R> object_size(dm_large)
   8.07 MB
   R> dm_large <- as.matrix(dist(1:10000))
   R> object_size(dm_large)
   801 MB

   If we did

   R> dm_large <- as.matrix(dist(1:1000000))

   we'd expect object_size(dm_large) to be around 80 GB, which should crash your personal computer's R/Rstudio since you probably don't have that much RAM.

   
   
   

   It is possible to also check your amount of RAM usage with OS-centric tools (e.g. Mac's: Activity Monitor or Window's: Dask Management).