Stat DS Computing Wiki - User contributions [en]

Linux and department servers

2025-06-12T15:22:36Z

Jwalchessen: passwordless ssh (Julia)

The department has a set of servers, the hydras, that can help you run large analyses and gain a better understanding of common command line structure for more high scale computing. One can also view the department servers a bit more as a “sandbox” or “playground” as there are very few constraints and it’s okay to not run as polished code as you might on the [[PSC]] or Azure.

['''Request:''' Windows users (and all users) - if you see comments that are outdated / less helpful than expected please contribute.]

== What resources exist ==

# The department provides 13 servers (named {<code>hydra1</code>, ... , <code>hydra13</code>}<code>.stat.cmu.edu</code>.
# Each hydra has 12 CPU cores, and 64 GB of RAM (per core).
## The exceptions are hydra7 and hydra8, which have 128 GB of RAM. Use these if you have particularly large jobs that need lots of memory.
# The hydras all share the same data file system. This allows you to upload a file to any of the hydras and then have it on all of them.
# [[OpenMPI]] is installed
#* This allows MPI based parallelization to be used
#* Note that the hydras ARE NOT network connected, so MPI can only be used within a single hydra. It cannot communicate across multiple hydras.
#* If you need to run something in parallel across multiple nodes, you should use [[PSC]] instead.

''See [[what_resources_when| What resources when]] for guides to help decide which resource you should be using.''

== Using the servers respectfully ==
Because the servers are a shared department resource it is important to use them respectfully. There are many servers and multiple cores (CPUs) per server. To be respectful of other users:

# '''Check the current usage''' of the server you're on before using it.
# Do not use '''all''' the cores on a server, and do not use '''all''' the servers at the same time, so others can get work done.
# If you have a long-running job, '''deprioritize it''' so others can run too.

Each of these is easy to check.

=== Checking current usage ===
To check which servers are currently being used and which are open, see the [https://info.stat.cmu.edu/grafana/d/fdkb6l4klj400d/hydra-cluster?orgId=1 Grafana dashboard for the hydras]. It lists them all and their current usage.

When you're logged in to one node, you can use the <code>top</code> command. <code>top</code> displays the current usage of the server’s resources with a list of jobs run. It also tells you the percentage of the cores that are being used and the amount of RAM (memory) that is available. See [https://www.howtogeek.com/668986/how-to-use-the-linux-top-command-and-understand-its-output/ howtogeek] resources on how to interpret it all (the documentation is very clean, so we encourage you to read up ther

=== Limiting the cores you use ===
We recommend that when using parallel processing, use fewer than the total number of cores on the server. Most parallel processing packages let you choose the number of cores to use; the hydras have 12, so pick something smaller.

=== Deprioritize when possible ===
<code>nice/renice</code> allows you to lower the priority of your jobs, so the operating system lets other processes run first. See [https://www.howtogeek.com/411979/how-to-set-process-priorities-with-the-nice-and-renice-commands-in-linux/ howtogeek] resources on the basics. <code>nice</code> sets the priority, and <code>renice</code> updates your priority. On the department servers, everyone starts at the same priority, and you can only '''deprioritize''' your jobs -- you can't make them higher priority than everyone else.<syntaxhighlight lang="bash">nice -20 Rscript some-big-script.R
renice -n 20 processid # get the process ID from top</syntaxhighlight>
== Introduction to connecting to the server and useful Linux commands ==

The department servers all run Linux, and require you to use a command line to interact with it (as compared to Rstudio and other GUIs).
=== Getting Access: (<code>ssh</code>) ===

To get onto the server and type code you’ll need to access it through <code>ssh</code>.

<ul>
<li>For Macs (and Linux), open up a <code>terminal</code> and type
<syntaxhighlight lang="bash">ssh andrew_id@hydra1.stat.cmu.edu</syntaxhighlight>

you will then have to type in your password (if you don’t know it or don’t know it or have never accessed the server before - email [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu]).
</li>
<li>
For Windows, there's a few options (with different pros and cons). [https://www.makeuseof.com/tag/4-easy-ways-to-use-ssh-in-windows/ Makeuseof] provides an overview and introduction to how to get started on a few approaches, including PuTTY and Windows PowerShell.

Again, make sure to email [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu] to check that you have an account / or what your password is if you forgot.
</li>
</ul>
If you’re accessing the servers a lot, you might also look into aliases and password storing for the <code>ssh</code> commands. - Mac/Linux: [https://scotch.io/tutorials/how-to-create-an-ssh-shortcut scotch.io] - Windows: Powershell: [https://docs.microsoft.com/en-us/windows/terminal/tutorials/ssh microsoft], PuTTY: [https://www.makeuseof.com/tag/4-easy-ways-to-use-ssh-in-windows/ makeuseof]

=== Passwordless ssh ===
To get on the server without typing in the password, you need to first generate a ssh key pair.

* For Linux, open up a terminal and check first that the specific ssh key pair does not exist in the given folder.
* Next, generate a key pair (both public and private key). There are different types of keys (RSA, DSA, ECDSA, and ED25519). The -t option specifies the type of key pair.

<code>ssh-keygen -t rsa</code>

* A prompt for what to name the key pair and what folder to place it in should appear. The username should be the username for your local computer and id_rsa is the key pair name. You can change it. The local key should be stored in the .ssh folder on your local computer.

<code>Generating public/private rsa key pair.</code>

<code>Enter file in which to save the key (/home/username/.ssh/id_rsa):</code>

* Enter the folder and name for the key pair and hit enter.
* A prompt for a passphrase should appear. This is for securing the local key (on local computer) and is optional.

<code>Enter Passphrase (empty for no passphrase):</code>

* Make sure there is a .ssh folder on the virtual machine. If not, make a .ssh folder.
* Next, upload public key to remote server. This command will upload the public key to all the hydras and ghidorahs since they share a file system.

<code>ssh-copy-id andrew_id@hydra1.stat.cmu.edu</code>

* To ensure the permissions of the .ssh directory are updated:

<code>chmod 700 ~/.ssh</code>

<code>chmod 600 ~/.ssh/authorized_keys</code>

You should now be able to ssh into the server without being prompted for a password.

=== File sharing (<code>scp</code> +) ===

There is also command line code to copy files from your personal computer to the server. This is done with a SCP (secure copy) tool. There are also GUIs (graphical user interfaces) that allow you to “drag and drop” files from your computer to the server (and visa-versa). Currently, a lot of students use GUIs to deal with file sharing.

==== Command line <code>scp</code>====

You can file share either by using <code>scp</code>, for example

<ol>
<li> Mac/Linux: This command is very similar to <code>cp</code> (the file copier).
<ol>
<li>Copy from server
<syntaxhighlight lang="bash"> scp username@from_host:path/to/file.txt /local/directory/ </syntaxhighlight>
<syntaxhighlight lang="bash">scp -r username@from_host:path/to/folder/ local/directory/</syntaxhighlight>
</li>
<li>Copy to server
 
<syntaxhighlight lang="bash"> scp local/file.txt username@to_host:/remote/directory/</syntaxhighlight>
<syntaxhighlight lang="bash"> scp -r local/directory/ username@to_host:/remote/directory/</syntaxhighlight>
</li>
</ol>

<code>scp</code> references: [https://haydenjames.io/linux-securely-copy-files-using-scp/ haydenjames.io]

</li>
<li>Windows: Again, this process is a bit more complicated.
<ol>
<li>A PuTTY + Command Prompt provides a version that is very similar to Mac/Linux <code>scp</code> (reference: [https://success.tanaza.com/s/article/How-to-use-SCP-command-on-Windows tanaza]), and requires minor changes in the call  (<code>pscp -scp</code> instead of <code>scp)</code>. </li>
<li>A Powershell specific command is a bit more complicated / not similar to Mac/Linux <code>scp</code> (reference: [https://www.nsoftware.com/kb/articles/psserver-scp.rst nsoftware]) </li>
</ol>
</li>
</ol>

===  GUI “drag-and-drop” file sharing ===
A '''lot''' of students use '''GUI''' interfaces to transfer files instead of <code>scp</code> command line. These tools allow you to “drag and drop” files from one location to another after the connection is set up.

Some examples include:

<ol>
<li>Mac:
<ol><li>
[https://cyberduck.io/ Cyberduck]
</li></ol></li>
<li>Linux:
<ol><li>
[https://www.openssh.com/ OpenSSH]
</li></ol></li>
<li> Windows: lots of options - [https://www.thegeekstuff.com/2011/06/windows-sftp-scp-clients/ thegeekstuff]
<ol>
<li>[https://www.putty.org/ PuTTY] </li>
<li>[https://winscp.net/eng/download.php WinSCP] </li></ol>
</li>
</ol>
=== Basic CLI (Linux command line): <code>bash</code>===

Understanding the basic Linux command line is important to be able to navigate when you are actually on the server. Below are some based commands to move around the file system, manipulate files, and view storage structure.

{| class="wikitable"
|-
! command
! translation
|-
| <code>pwd</code>
| print the current working directory
|-
| <code>cd _dir_</code>
| change directory
|-
| <code>rm _file_</code>
| remove / delete file
|-
| <code>mv _file_ _dir_</code>
| move file (can change name)
|-
| <code>cp _file_ _dir_</code>
| copy file
|-
| <code>ls _dir_</code>
| list files in directory
|}

use <code>man _command_</code> to get a listing / manual of the command.

Additional resources that capture more information on the command line and are still geared to statistical computing can be found here: [http://www.jarrodmillman.com/rcsds/standard/bash.html UCB 2015 reproducible DS lecture notes].

'''TODO: add information about <code>sed</code>, <code>chmod</code>, and a link to bash files.'''

== Text editors ==

There are multiple options to view code on the server side. They include:

*vim
*… [to be added later]

== Running jobs ==

We have included thoughts about running jobs in two places.

#The first looks at tools that allow you to keep running jobs on the server when you close the connection to the server (turn off your laptop, etc.). These tools are call [[Terminal_multiplexers|terminal multiplexers]]. 
#The second set of tools focus on how to run your [[R_on_the_server|R analysis]] or [[Python_on_the_server|Python analysis]] code on the server.

== Server specs ==

Generally speaking, our servers have multiple CPU cores, but no GPU cores. For deep learning tasks that need GPUs, consider using the [[GPU Cluster|GPU cluster]].

To examine the specs yourself, use
<syntaxhighlight lang="bash">cat /proc/meminfo
cat /proc/cpuinfo</syntaxhighlight>

to see some information.

Main Page

2025-05-22T18:21:50Z

Jwalchessen: /* Computing Office Hours */

Anyone can read this wiki.  If you want to write/edit stuff in the wiki, you will need to log in with your account provided by the Statistics & Data Science Department computing group. This Wiki is intended to be a repository for instruction, information and ideas related to the many computing environments available to members and affiliates of the Statistics & Data Science Department.

== Getting to know the computing resources & getting support ==

If you're interested in knowing about what resources you have access to (and/or) what resources you have access to that might solve a problem you're having, please visit our [[What_resources_when|"what resources when" article]].

To get support that is beyond this website please attend office hours below and/or email [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu] for support (for more details about how the support ticketing works through [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu] + ServiceNow please see the [[Service_now|ServiceNow]] article).

Reset your StatDS password here - https://util.stat.cmu.edu/passwordreset/

== Department Contacts ==

• [mailto:events@stat.cmu.edu events@stat.cmu.edu]: seminars, conferences & workshops, and calendar updates

• [mailto:news@stat.cmu.edu news@stat.cmu.edu]: dept/dept member-related public news & announcements

• [mailto:lists@stat.cmu.edu lists@stat.cmu.edu]: department email lists

• [mailto:webmaster@stat.cmu.edu webmaster@stat.cmu.edu]: website updates

• [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu]: computer/tech requests

• [mailto:rooms@stat.cmu.edu rooms@stat.cmu.edu]: request space for meetings and TA office hours

• [mailto:reimbursements@stat.cmu.edu reimbursements@stat.cmu.edu]: travel and reimbursement requests

== Computing Office Hours ==

These office hour opportunities provide a dedicated time for you to come ask questions about research computing.  These include code advice, how to access certain resources, what the best resources are for your project, or to learn about new connections to help you progress in your education and research. Staff Office hour times will be sent out each week on Monday as a calendar invite; accept any of the invitations to add it to your personal calendar. PhD TA Office Hours are below, unless a different time is communicated.

#PhD TA Office Hours for the first half of summer 2025 are held by Julia Walchessen in FMS Room 320 on Mondays from 2:30 pm to 4:30 pm
#Staff Office Hours are held by appointment by Carl Skipper in BH 129G. Schedule via [mailto:lists@stat.cmu.edu email].

See [[Computing_office_hour_usage|computing office hour usage]] for what office hours can be used for.

== Computing Wiki Pages ==

=== Managing servers and environments ===

==== Virtual environments ====
*[[Virtual Environment Introduction|Why use virtual environments?]]
*R's [[Renv]]
*Python's [[Python_virtual_environments|virtual environments (2 options)]]
*Old instructions for [[Conda with Python (Reference Only)|conda with Python]]

==== Containers ====
*[[Introduction to containers]]
*[[Docker, and why-you-should-use-it-for-your-research]]
*[[Tutorials|Basic tutorial]]
*[[Running Docker locally and on Hydras]]

=== Department computing resources ===

==== Printing ====
*[[Department_Printers|Department Printers]]

==== Web Hosting ====
#[[How to host your webpage under stat.cmu.edu]]
#[[Bulding your website with Hugo]]

==== Servers ====
*[[Linux_and_department_servers|Department Servers]]. This wiki introduces you to the '''hydra''' servers, basic Linux commands, and more.
*#[[Terminal_multiplexers|Terminal multiplexers (screen and tmux)]]. This wiki presents tools for use on the ''hydra'' that can allow you to run jobs even when your not on the server, and much much more!
*#[[Python on the server]]. This wiki introduces uses command line approaches to running <code>python</code> on a virtual machine, and how to use jupyter notebooks through the server as well.
*#[[R on the server]]. This wiki introduces uses command line approaches to running <code>R</code> on a virtual machine

*[[Department GPU Servers]], for small jobs needing 1 GPU.
*[[GPU Cluster]], for larger jobs that may need multiple GPUs and many CPUs.
*[[PSC]], the Pittsburgh Supercomputing Center. Free accounts and usage credits are available
*#[[ACCESS]]
*#[[Slurm]]. This wiki introduces the job management system uses at PSC.
*#[[Parallelization_on_PSC/Bridges-2]]
*#[[Access PSC compute nodes directly from VSCode]]
*Azure
*#[[Azure_getting_started|Azure: getting started]]. This wiki walks you through the steps to create a virtual machine through Azure
*#[[How to obtain a virtual machine with a GPU]]
*#[[Attach a disk to a virtual machine]]

=== Development guides ===

====[[Parallel_computing|Parallelization]]====
*[[Parallelization_in_r|R-specific parallelization]]
*[[Parallelization_in_python|Python-specific parallelization]]

==== Package development ====
*[[Developing_R_packages|R package development]]
*[[Developing_python_packages|Python package development]]
[[Terminal automation]]. This wiki introduces approaches to automate Linux/OS command line calls.
*[[Bash files]]. This wiki presents potential uses of Bash files including automated file/folder creation, parallelization and much more!
*[[Make]]. This wiki presents potential uses of <code>make</code> and Makefiles including automated workflow and dependency tracking
*[[Jumping to frequently used folders and automating environment setups]]

==== Version Control ====
See the guide provided in the site of [https://36-750.github.io/tools/version-control/ 36750 - Statistical Computing]

==== Reference management ====
#Zotero
#Mendeley

==== Integrated Development Environments (IDEs) ====
#[[VSCode|VSCode]]
#PyCharm
#RStudio
#Eclipse

==== Text Editors ====
#Vim
#Nano
#Emacs
#Atom
#Sublime

See also the guide provided in the site of [https://36-750.github.io/tools/editors/ 36750 - Statistical Computing]

==== AI Tools ====
* [[ChatGPT and other LLMs]]

*[[github_copilot|GitHub Copilot]]

==== Miscellanea ====
#[[Windows_Subsystem_for_Linux_(WSL)_and_Python|Windows Subsystem for Linux (WSL) and Python]]
#[[Amphetamine|Amphetamine: How to keep your computer awake]]

=== General Department Computing Resources ===

*Fall 2021 [[Computing_seminar_resources|Computing Seminar Resources]] (video and slides)

== Ideas for the Wiki ==

Please see the [[TODO|TODO page]] to post suggestions and errata.

== Using the Wiki ==

*Here is some [[Generic_Wiki_Information|Generic Wiki Information]] that you may be interested in, such as a users' guide and other resources for Wiki users.

Main Page

2025-05-22T18:21:20Z

Jwalchessen: /* Computing Office Hours */

Anyone can read this wiki.  If you want to write/edit stuff in the wiki, you will need to log in with your account provided by the Statistics & Data Science Department computing group. This Wiki is intended to be a repository for instruction, information and ideas related to the many computing environments available to members and affiliates of the Statistics & Data Science Department.

== Getting to know the computing resources & getting support ==

If you're interested in knowing about what resources you have access to (and/or) what resources you have access to that might solve a problem you're having, please visit our [[What_resources_when|"what resources when" article]].

To get support that is beyond this website please attend office hours below and/or email [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu] for support (for more details about how the support ticketing works through [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu] + ServiceNow please see the [[Service_now|ServiceNow]] article).

Reset your StatDS password here - https://util.stat.cmu.edu/passwordreset/

== Department Contacts ==

• [mailto:events@stat.cmu.edu events@stat.cmu.edu]: seminars, conferences & workshops, and calendar updates

• [mailto:news@stat.cmu.edu news@stat.cmu.edu]: dept/dept member-related public news & announcements

• [mailto:lists@stat.cmu.edu lists@stat.cmu.edu]: department email lists

• [mailto:webmaster@stat.cmu.edu webmaster@stat.cmu.edu]: website updates

• [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu]: computer/tech requests

• [mailto:rooms@stat.cmu.edu rooms@stat.cmu.edu]: request space for meetings and TA office hours

• [mailto:reimbursements@stat.cmu.edu reimbursements@stat.cmu.edu]: travel and reimbursement requests

== Computing Office Hours ==

These office hour opportunities provide a dedicated time for you to come ask questions about research computing.  These include code advice, how to access certain resources, what the best resources are for your project, or to learn about new connections to help you progress in your education and research. Staff Office hour times will be sent out each week on Monday as a calendar invite; accept any of the invitations to add it to your personal calendar. PhD TA Office Hours are below, unless a different time is communicated.

#PhD TA Office Hours are held by Julia Walchessen in FMS Room 320 on Mondays from 2:30 pm to 4:30 pm
#Staff Office Hours are held by appointment by Carl Skipper in BH 129G. Schedule via [mailto:lists@stat.cmu.edu email].

See [[Computing_office_hour_usage|computing office hour usage]] for what office hours can be used for.

== Computing Wiki Pages ==

=== Managing servers and environments ===

==== Virtual environments ====
*[[Virtual Environment Introduction|Why use virtual environments?]]
*R's [[Renv]]
*Python's [[Python_virtual_environments|virtual environments (2 options)]]
*Old instructions for [[Conda with Python (Reference Only)|conda with Python]]

==== Containers ====
*[[Introduction to containers]]
*[[Docker, and why-you-should-use-it-for-your-research]]
*[[Tutorials|Basic tutorial]]
*[[Running Docker locally and on Hydras]]

=== Department computing resources ===

==== Printing ====
*[[Department_Printers|Department Printers]]

==== Web Hosting ====
#[[How to host your webpage under stat.cmu.edu]]
#[[Bulding your website with Hugo]]

==== Servers ====
*[[Linux_and_department_servers|Department Servers]]. This wiki introduces you to the '''hydra''' servers, basic Linux commands, and more.
*#[[Terminal_multiplexers|Terminal multiplexers (screen and tmux)]]. This wiki presents tools for use on the ''hydra'' that can allow you to run jobs even when your not on the server, and much much more!
*#[[Python on the server]]. This wiki introduces uses command line approaches to running <code>python</code> on a virtual machine, and how to use jupyter notebooks through the server as well.
*#[[R on the server]]. This wiki introduces uses command line approaches to running <code>R</code> on a virtual machine

*[[Department GPU Servers]], for small jobs needing 1 GPU.
*[[GPU Cluster]], for larger jobs that may need multiple GPUs and many CPUs.
*[[PSC]], the Pittsburgh Supercomputing Center. Free accounts and usage credits are available
*#[[ACCESS]]
*#[[Slurm]]. This wiki introduces the job management system uses at PSC.
*#[[Parallelization_on_PSC/Bridges-2]]
*#[[Access PSC compute nodes directly from VSCode]]
*Azure
*#[[Azure_getting_started|Azure: getting started]]. This wiki walks you through the steps to create a virtual machine through Azure
*#[[How to obtain a virtual machine with a GPU]]
*#[[Attach a disk to a virtual machine]]

=== Development guides ===

====[[Parallel_computing|Parallelization]]====
*[[Parallelization_in_r|R-specific parallelization]]
*[[Parallelization_in_python|Python-specific parallelization]]

==== Package development ====
*[[Developing_R_packages|R package development]]
*[[Developing_python_packages|Python package development]]
[[Terminal automation]]. This wiki introduces approaches to automate Linux/OS command line calls.
*[[Bash files]]. This wiki presents potential uses of Bash files including automated file/folder creation, parallelization and much more!
*[[Make]]. This wiki presents potential uses of <code>make</code> and Makefiles including automated workflow and dependency tracking
*[[Jumping to frequently used folders and automating environment setups]]

==== Version Control ====
See the guide provided in the site of [https://36-750.github.io/tools/version-control/ 36750 - Statistical Computing]

==== Reference management ====
#Zotero
#Mendeley

==== Integrated Development Environments (IDEs) ====
#[[VSCode|VSCode]]
#PyCharm
#RStudio
#Eclipse

==== Text Editors ====
#Vim
#Nano
#Emacs
#Atom
#Sublime

See also the guide provided in the site of [https://36-750.github.io/tools/editors/ 36750 - Statistical Computing]

==== AI Tools ====
* [[ChatGPT and other LLMs]]

*[[github_copilot|GitHub Copilot]]

==== Miscellanea ====
#[[Windows_Subsystem_for_Linux_(WSL)_and_Python|Windows Subsystem for Linux (WSL) and Python]]
#[[Amphetamine|Amphetamine: How to keep your computer awake]]

=== General Department Computing Resources ===

*Fall 2021 [[Computing_seminar_resources|Computing Seminar Resources]] (video and slides)

== Ideas for the Wiki ==

Please see the [[TODO|TODO page]] to post suggestions and errata.

== Using the Wiki ==

*Here is some [[Generic_Wiki_Information|Generic Wiki Information]] that you may be interested in, such as a users' guide and other resources for Wiki users.