View source for Linux and department servers

[[Category:Department Servers]]
[[Category:Remote Server Access]]

{{ScalingSidebar}}

The department has two sets of servers, the Leviathans and the Hydras, that can help you run large analyses and gain a better understanding of common command line structure for more high scale computing. 

<div style="background-color: #e8f4f8; border-left: 5px solid #17a2b8; padding: 15px; margin-bottom: 20px; margin-top: 15px; overflow: hidden;">
'''Important: The Sandbox Philosophy'''<br>
Think of the department servers as a “sandbox” or “playground.” There are very few constraints here, making it a great place to learn. It is perfectly fine to run code here that isn't quite polished enough for strict cluster environments like the [[PSC]] or [[Wright GPU Cluster]].
</div>

['''Request:''' All users - if you see comments that are outdated / less helpful than expected please contribute.]

== What Resources Exist ==

# The department provides 11 CPU servers (named <code>leviathan1.stat.cmu.edu</code> through <code>leviathan11.stat.cmu.edu</code>).
# The department provides 3 GPU servers (named <code>ghidorah1.stat.cmu.edu</code> through <code>ghidorah3.stat.cmu.edu</code>).
# The Ghidorahs and Leviathans all share the same network-attached data file system. This allows you to upload a file to any single server and instantly access it on all of them.

''See [[what_resources_when| What resources when]] for guides to help decide which resource you should be using.''

== Getting Access (SSH) ==

The department servers all run Linux and require you to interact with them remotely. To execute code, you’ll need to access them through <code>ssh</code>. Modern operating systems (Mac, Linux, and Windows 10/11) have this tool built-in natively.

* '''Mac / Linux:''' Open up your <code>Terminal</code>.
* '''Windows:''' Open up <code>Windows PowerShell</code> or <code>Windows Terminal</code>.

In your terminal, type: 
<syntaxhighlight lang="bash">ssh andrew_id@leviathan1.stat.cmu.edu</syntaxhighlight>

You will be prompted to type in your password. (Note: as you type, no characters will appear on the screen for security reasons. Just type it and hit Enter). If you don’t know your password or have never accessed the server before, email [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu].

=== Passwordless SSH (Highly Recommended) ===
To connect to the server without typing in your password every time, you need to generate an SSH key pair. 

# Check whether you already have a key pair by looking in your <code>.ssh</code> folder (run <code>ls ~/.ssh</code> on Mac/Linux, or <code>dir ~\.ssh</code> on Windows). If you see <code>id_ed25519</code> or <code>id_rsa</code>, you can skip to step 3.
# To generate a new key pair, run: <syntaxhighlight lang="bash">ssh-keygen -t ed25519</syntaxhighlight> Hit Enter to accept the default file location. You can enter a passphrase for extra security or leave it empty for true passwordless login.
# Upload the public key to the remote server (you only need to do this once since the servers share a file system).
#* '''For Mac/Linux:''' <br><code><syntaxhighlight lang="bash">ssh-copy-id andrew_id@leviathan1.stat.cmu.edu</syntaxhighlight></code>
#* '''For Windows:''' Windows does not natively include <code>ssh-copy-id</code>. Use this PowerShell one-liner instead:<br><code><syntaxhighlight lang="powershell">Get-Content "$env:USERPROFILE\.ssh\id_ed25519.pub" | ssh andrew_id@leviathan1.stat.cmu.edu "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys"</syntaxhighlight></code>
# Finally, log into the server normally and run the following to enforce strict permissions:
# <syntaxhighlight lang="bash">
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
</syntaxhighlight>

== File Sharing & Management ==

There are two primary ways to move files between your personal computer and the server.

=== Command Line (<code>scp</code>) ===
The <code>scp</code> (secure copy) command works identically across Mac, Linux, and Windows PowerShell.

{| style="width: 100%; border-collapse: collapse; margin-bottom: 10px;"
|-
! style="width: 50%; background-color: #d4edda; color: #155724; padding: 10px; text-align: left;" | Strengths
! style="width: 50%; background-color: #f8d7da; color: #721c24; padding: 10px; text-align: left;" | Weaknesses
|- style="vertical-align: top; background-color: #f8f9fa;"
| style="padding: 15px; border-right: 2px solid white;" |
* '''Fast and Native:''' Requires no additional software installations.
* '''Scriptable:''' Can be built into bash scripts to automate backing up data.
| style="padding: 15px;" |
* '''Syntax Heavy:''' Easy to forget the exact command structure or path names.
* '''No Visual Feedback:''' Doesn't offer a traditional file-explorer view of the remote server.
|}

'''From server to your computer:'''
<syntaxhighlight lang="bash">scp andrew_id@leviathan1.stat.cmu.edu:/path/to/remote/file.txt /local/directory/</syntaxhighlight>
*(To copy an entire folder, add the <code>-r</code> flag right after <code>scp</code>).*

'''From your computer to the server:'''
<syntaxhighlight lang="bash">scp /local/file.txt andrew_id@leviathan1.stat.cmu.edu:/remote/directory/</syntaxhighlight>

=== GUI "Drag-and-Drop" Tools ===
Many students prefer graphical interfaces to transfer files.

* '''Visual Studio Code (Recommended):''' If you use the VS Code Remote-SSH extension, you can simply drag and drop files from your laptop directly into the VS Code file explorer pane.
* '''Mac:''' [https://cyberduck.io/ Cyberduck] remains a solid dedicated tool.
* '''Windows:''' [https://winscp.net/eng/download.php WinSCP] is a popular dedicated GUI (legacy SSH clients like PuTTY are no longer necessary).

== Basic Linux Command Line (Bash) ==

Understanding the basic Linux command line is crucial for navigating the servers. Use <code>man _command_</code> to get the manual for any command.

{| class="wikitable" style="width: 100%; text-align: left;"
|-
! Command !! Translation !! Example Usage
|-
| <code>pwd</code> || Print working directory || Shows exactly where you are in the file system.
|-
| <code>ls _dir_</code> || List files || <code>ls -l</code> shows file permissions and sizes.
|-
| <code>cd _dir_</code> || Change directory || <code>cd ..</code> moves you up one folder.
|-
| <code>rm _file_</code> || Remove / delete file || <code>rm -r _folder_</code> deletes an entire directory. '''(Irreversible!)'''
|-
| <code>mv _file_ _dir_</code> || Move / rename file || <code>mv data.csv ./cleaned_data.csv</code>
|-
| <code>cp _file_ _dir_</code> || Copy file || <code>cp script.R backup_script.R</code>
|-
| <code>chmod _perms_ _file_</code> || Change permissions || <code>chmod +x script.sh</code> makes a script executable.
|-
| <code>sed</code> || Stream editor || <code>sed -i 's/old/new/g' file.txt</code> replaces text inside a file automatically.
|}

For a deeper dive into shell scripting and Linux environments, we recommend the [http://www.jarrodmillman.com/rcsds/standard/bash.html UCB reproducible DS lecture notes] or the [https://tldp.org/LDP/Bash-Beginners-Guide/html/ Linux Documentation Project Bash Guide].

== Text Editors and IDEs for writing code on the servers==

Historically, editing code on a server required terminal-based text editors. While still useful, modern tools allow you to use a full graphical IDE on your laptop that computes natively on the server.

=== Visual Studio Code (Highly Recommended) ===
[https://code.visualstudio.com/ Visual Studio Code] is the industry standard for remote development. 

# Install VS Code on your personal machine.
# Go to the Extensions tab and install '''Remote - SSH'''.
# Click the green remote indicator in the bottom-left corner, select "Connect to Host...", and type <code>andrew_id@leviathan1.stat.cmu.edu</code>.
# Once connected, you can edit scripts, run interactive terminal sessions, and drag-and-drop files exactly as if they were on your laptop.

=== Terminal Editors ===
If you only need to make a quick tweak to a file while already logged into the terminal:

{| style="width: 100%; border-collapse: collapse; margin-bottom: 10px;"
|-
! style="width: 50%; background-color: #f8f9fa; border: 1px solid #dee2e6; padding: 10px; text-align: left;" | nano
! style="width: 50%; background-color: #f8f9fa; border: 1px solid #dee2e6; padding: 10px; text-align: left;" | vim
|- style="vertical-align: top;"
| style="padding: 15px; border: 1px solid #dee2e6;" |
A very simple, beginner-friendly text editor. Commands are listed at the bottom of the screen.
* Type <code>nano _filename_</code> to open.
* Use <code>Ctrl+O</code> to save (Write Out).
* Use <code>Ctrl+X</code> to exit.
| style="padding: 15px; border: 1px solid #dee2e6;" |
A highly powerful, ubiquitous editor with a steep learning curve.
* Type <code>vim _filename_</code> to open.
* Press <code>i</code> to enter Insert Mode (to type).
* Press <code>Esc</code> then type <code>:wq</code> to save and quit.
|}

== Using the Servers Respectfully ==

Because the servers are a shared department resource, it is critical to use them considerately.

<div style="background-color: #fff3cd; border-left: 5px solid #ffc107; padding: 15px; margin-bottom: 20px; margin-top: 15px;">
'''The Golden Rules of Shared Servers:'''<br>
1. '''Check current usage''' before launching massive scripts.<br>
2. '''Do not use all the cores''' on a single server.<br>
3. '''Do not use all the servers''' at the exact same time.<br>
4. '''Deprioritize long jobs''' so active users aren't frozen out.
</div>

=== 1. Checking Current Usage ===
Check the [https://util.stat.cmu.edu/grafana/d/beuw1q6y3dgjkc/hydra-cluster?orgId=1&from=now-6h&to=now&timezone=browser Grafana dashboard for the Hydras] and the [https://util.stat.cmu.edu/grafana/d/asjkldfbnvajdsfh213/leviathan-cluster?orgId=1&from=now-6h&to=now&timezone=browser Grafana dashboard for the Leviathans] to see active usage across all nodes.

Once logged into a specific node, run the <code>top</code> command. This displays the percentage of cores being used, available RAM, and a list of running jobs. Press <code>q</code> to exit <code>top</code>. (See [https://www.howtogeek.com/668986/how-to-use-the-linux-top-command-and-understand-its-output/ this HowToGeek guide] for interpreting the output).

=== 2. Limiting Your Cores ===
When using parallel processing packages, explicitly set the number of cores to less than the server's maximum. For example, the hydras have 12 cores—hardcode your script to use 8 or fewer so the OS has room to breathe.

=== 3. Deprioritize When Possible (nice/renice) ===
<code>nice</code> allows you to lower the priority of your jobs, telling the OS to let other processes run first. Everyone starts at the same priority level, and you can only '''deprioritize''' your own jobs.

<syntaxhighlight lang="bash">
# Start a new job with lowered priority
nice -20 Rscript some-big-script.R

# Lower the priority of a job that is already running
renice -n 20 processid # get the process ID from the 'top' command
</syntaxhighlight>

=== 4. Using <code>/scratch</code> for Heavy Data ===
<div style="background-color: #f8d7da; border-left: 5px solid #dc3545; padding: 15px; margin-bottom: 20px; margin-top: 15px; overflow: hidden;">
'''CRITICAL: Do not run heavy Read/Write jobs from your Home directory!'''<br>
Your home directory lives on a central network-attached storage (NAS) drive. If you read/write massive datasets directly to <code>~</code>, you will bottleneck the network and slow down operations (even basic SSH access) for '''every single user across all servers'''.
</div>

Instead, use the <code>/scratch</code> folder. This folder points to a local, lightning-fast SSD specific to the server you are currently logged into. 

'''How to use <code>/scratch</code>:'''
<syntaxhighlight lang="bash">
cd /scratch
mkdir andrew_id_data # Create a folder for yourself
cd andrew_id_data
# Move your data here and run your heavy I/O scripts!
</syntaxhighlight>

'''Important Caveats for <code>/scratch</code>:'''
* It is specific to the node you are on (<code>leviathan1</code>'s scratch is different from <code>leviathan2</code>'s scratch). 
* It is strictly for temporary runtime storage. '''Do not use it for long-term storage.'''
* Space is limited. Delete your files when your script finishes running.

== Running Jobs & Multiplexers ==

If you start a script over standard SSH and close your laptop, the connection breaks and your job is instantly killed. 

To run jobs that persist after you disconnect, you must use a '''terminal multiplexer''' like <code>tmux</code> or <code>screen</code>. These tools create a virtual session on the server that keeps running in the background.

* Start a new session: <code>tmux new -s my_analysis</code>
* Detach (leave it running in background): Press <code>Ctrl+b</code>, release, then press <code>d</code>
* Reattach later: <code>tmux attach -t my_analysis</code>

For language-specific guides on running batch analyses, see our dedicated pages for [[R_on_the_server|R analysis]] and [[Python_on_the_server|Python analysis]]. For more advanced multiplexer features, see [[terminal multiplexers]].

== Server Specs ==

Our servers have multiple CPU cores, but only the ghidorah servers have GPU cores. For deep learning tasks that require GPUs, consider using the [[GPU Cluster]].

To check the hardware specifications of the server you are currently logged into, run:
<syntaxhighlight lang="bash">
cat /proc/cpuinfo # View processor details and core counts
cat /proc/meminfo # View RAM and memory allocation
</syntaxhighlight>