Linux and department servers: Difference between revisions

From Stat DS Computing Wiki
Newer edit →
VisualWikitext
Created page with " ['''Request:''' Windows users (and all users) - if you see comments that are outdated / less helpful than expected please contribute.] = Introduction (why use the department..."
 
fix /proc/ typos, clean up headings
Line 2: Line 2:
['''Request:''' Windows users (and all users) - if you see comments that are outdated / less helpful than expected please contribute.]
['''Request:''' Windows users (and all users) - if you see comments that are outdated / less helpful than expected please contribute.]


= Introduction (why use the department servers / philosophical use) =
== Introduction (why use the department servers / philosophical use) ==


The department has a set of servers (hosted virtually through campus computing) that can help you process analysis faster and gain a better understanding of common command line structure for more high scale computing. One can also view the department servers a bit more as a “sandbox” or “playground” as there are very few constraints and it’s ok to not run as polished code as you might on the PSC or Azure.
The department has a set of servers (hosted virtually through campus computing) that can help you process analysis faster and gain a better understanding of common command line structure for more high scale computing. One can also view the department servers a bit more as a “sandbox” or “playground” as there are very few constraints and it’s ok to not run as polished code as you might on the PSC or Azure.


= What resources exist =
== What resources exist ==


# The department provides 13 servers (named {<code>hydra1</code>, ... , <code>hydra13</code>}<code>.stat.cmu.edu</code>.  
# The department provides 13 servers (named {<code>hydra1</code>, ... , <code>hydra13</code>}<code>.stat.cmu.edu</code>.  
Line 15: Line 15:
''See [[what_resources_when| What resources when]] for guides to help decide which resource you should be using.''
''See [[what_resources_when| What resources when]] for guides to help decide which resource you should be using.''


= Introduction to connecting to the server and useful Linux commands =
== Introduction to connecting to the server and useful Linux commands ==


The department servers all run Linux, and require you to use a command line to interact with it (as compared to Rstudio and other GUIs).
The department servers all run Linux, and require you to use a command line to interact with it (as compared to Rstudio and other GUIs).
 
=== Getting Access: (<code>ssh</code>) ===
 
== Getting Access: (<code>ssh</code>) ==


To get onto the server and type code you’ll need to access it through <code>ssh</code>.
To get onto the server and type code you’ll need to access it through <code>ssh</code>.
Line 38: Line 36:
If you’re accessing the servers a lot, you might also look into aliases and password storing for the <code>ssh</code> commands. - Mac/Linux: [https://scotch.io/tutorials/how-to-create-an-ssh-shortcut scotch.io] - Windows: Powershell: [https://docs.microsoft.com/en-us/windows/terminal/tutorials/ssh microsoft], PuTTY: [https://www.makeuseof.com/tag/4-easy-ways-to-use-ssh-in-windows/ makeuseof]
If you’re accessing the servers a lot, you might also look into aliases and password storing for the <code>ssh</code> commands. - Mac/Linux: [https://scotch.io/tutorials/how-to-create-an-ssh-shortcut scotch.io] - Windows: Powershell: [https://docs.microsoft.com/en-us/windows/terminal/tutorials/ssh microsoft], PuTTY: [https://www.makeuseof.com/tag/4-easy-ways-to-use-ssh-in-windows/ makeuseof]


== File sharing (<code>scp</code> +) ==
=== File sharing (<code>scp</code> +) ===


There is also command line code to copy files from your personal computer to the server. This is done with a SCP (secure copy) tool. There are also GUIs (graphical user interfaces) that allow you to “drag and drop” files from your computer to the server (and visa-versa). Currently, a lot of students use GUIs to deal with file sharing.
There is also command line code to copy files from your personal computer to the server. This is done with a SCP (secure copy) tool. There are also GUIs (graphical user interfaces) that allow you to “drag and drop” files from your computer to the server (and visa-versa). Currently, a lot of students use GUIs to deal with file sharing.


=== Command line <code>scp</code> ===
==== Command line <code>scp</code>====


You can file share either by using <code>scp</code>, for example
You can file share either by using <code>scp</code>, for example
Line 75: Line 73:
&nbsp;
&nbsp;


=== Command line <code>sftp</code> ===
==== Command line <code>sftp</code>====


[to do: add section on sftp]
[to do: add section on sftp]


=== GUI “drag and drop” file sharing ===
==== GUI “drag and drop” file sharing ====


A '''lot''' of students use '''GUI''' interfaces to transfer files instead of <code>scp</code> command line. These tools allow you to “drag and drop” files from one location to another after the connection is set up.
A '''lot''' of students use '''GUI''' interfaces to transfer files instead of <code>scp</code> command line. These tools allow you to “drag and drop” files from one location to another after the connection is set up.
Line 104: Line 102:




== Basic CLI (Linux command line): <code>bash</code> ==
=== Basic CLI (Linux command line): <code>bash</code>===


Understanding the basic Linux command line is important to be able to navigate when you are actually on the server. Below are some based commands to move around the file system, manipulate files, and view storage structure.
Understanding the basic Linux command line is important to be able to navigate when you are actually on the server. Below are some based commands to move around the file system, manipulate files, and view storage structure.
Line 155: Line 153:
== Server specs ==
== Server specs ==


Generally speaking, our servers have multiple CPU cores, but no GPU cores (as such, they are not the best for deep learning).
Generally speaking, our servers have multiple CPU cores, but no GPU cores. For deep learning tasks that need GPUs, consider using the [[GPU Cluster|GPU cluster]].


To examine the specs yourself, use
To examine the specs yourself, use
<syntaxhighlight lang="bash">cat proc/meminfo
<syntaxhighlight lang="bash">cat /proc/meminfo
cat proc/cpuinfo</syntaxhighlight>
cat /proc/cpuinfo</syntaxhighlight>


to see some information.
to see some information.
 
== Using the server respectfully ==
 
= Using the server respectively =


Because the servers are a shared department resource it is important to use them respectfully. There are many servers and multiple cores (CPUs) per server. To be respectful of other users we recommend checking (1) the current usage of any server you’re on (it’s possible one server is being used a lot but others are not), (2) not run a parallelize process that could use '''all''' of a servers cores (at least leave 1 core open), and (3) deprioritize your jobs that you don’t need to have done soon (especially if others are using the server when you submit your job).
Because the servers are a shared department resource it is important to use them respectfully. There are many servers and multiple cores (CPUs) per server. To be respectful of other users we recommend checking (1) the current usage of any server you’re on (it’s possible one server is being used a lot but others are not), (2) not run a parallelize process that could use '''all''' of a servers cores (at least leave 1 core open), and (3) deprioritize your jobs that you don’t need to have done soon (especially if others are using the server when you submit your job).
Line 173: Line 169:
</li>
</li>


<li>'''Parallelizing with less than 100% of the cores:'''<br/> To avoid using all the cores when parallelizing, we recommend you first look at <code>top</code>, and then at least do <code>(# of total cores) -1</code> as the upper limit of the number of cores used (less if others are using it as well). </li>
<li>'''Parallelizing with less than 100% of the cores:'''<br/>To avoid using all the cores when parallelizing, we recommend you first look at <code>top</code>, and then at least do <code>(# of total cores) -1</code> as the upper limit of the number of cores used (less if others are using it as well). </li>
<li>'''Deprioritizing when possible:'''<br/> <code>nice/renice</code> allows you to lower the priority of your jobs (so that others may use the resources if necessary) See [https://www.howtogeek.com/411979/how-to-set-process-priorities-with-the-nice-and-renice-commands-in-linux/ howtogeek] resources on the basics. <code>nice</code> sets the priority, and <code>renice</code> updates your priority. On the department servers, your can only '''deprioritize''' your jobs, everyone starts out on the same level.  
<li>'''Deprioritizing when possible:'''<br/> <code>nice/renice</code> allows you to lower the priority of your jobs (so that others may use the resources if necessary) See [https://www.howtogeek.com/411979/how-to-set-process-priorities-with-the-nice-and-renice-commands-in-linux/ howtogeek] resources on the basics. <code>nice</code> sets the priority, and <code>renice</code> updates your priority. On the department servers, your can only '''deprioritize''' your jobs, everyone starts out on the same level.  
<syntaxhighlight lang="bash">nice --20 ./andrew_id
<syntaxhighlight lang="bash">nice --20 ./andrew_id

Revision as of 16:11, 10 January 2025

[Request: Windows users (and all users) - if you see comments that are outdated / less helpful than expected please contribute.]

Introduction (why use the department servers / philosophical use)

The department has a set of servers (hosted virtually through campus computing) that can help you process analysis faster and gain a better understanding of common command line structure for more high scale computing. One can also view the department servers a bit more as a “sandbox” or “playground” as there are very few constraints and it’s ok to not run as polished code as you might on the PSC or Azure.

What resources exist

  1. The department provides 13 servers (named {hydra1, ... , hydra13}.stat.cmu.edu.
  2. Each hydra has 12 CPU cores, and 64 GB of RAM (per core).
  3. The hydras all share the same data file system. This allows you to upload a file to any of the hydras and then have it on all of them.
  4. OpenMPI is installed (which should allow parallelization across each server)

See What resources when for guides to help decide which resource you should be using.

Introduction to connecting to the server and useful Linux commands

The department servers all run Linux, and require you to use a command line to interact with it (as compared to Rstudio and other GUIs).

Getting Access: (ssh)

To get onto the server and type code you’ll need to access it through ssh.

  • For Macs (and Linux), open up a terminal and type 
    ssh andrew_id@hydra1.stat.cmu.edu

    you will then have to type in your password (if you don’t know it or don’t know it or have never accessed the server before - email remarks@stat.cmu.edu).

  • For Windows, there's a few options (with different pros and cons). Makeuseof provides an overview and introduction to how to get started on a few approaches, including PuTTY and Windows PowerShell. Again, make sure to email remarks@stat.cmu.edu to check that you have an account / or what your password is if you forgot.

If you’re accessing the servers a lot, you might also look into aliases and password storing for the ssh commands. - Mac/Linux: scotch.io - Windows: Powershell: microsoft, PuTTY: makeuseof

File sharing (scp +)

There is also command line code to copy files from your personal computer to the server. This is done with a SCP (secure copy) tool. There are also GUIs (graphical user interfaces) that allow you to “drag and drop” files from your computer to the server (and visa-versa). Currently, a lot of students use GUIs to deal with file sharing.

Command line scp

You can file share either by using scp, for example

  1. Mac/Linux: This command is very similar to cp (the file copier).
    1. Copy from server   
       scp username@from_host:path/to/file.txt /local/directory/
      scp -r username@from_host:path/to/folder/ local/directory/
    2. Copy to server   
       scp local/file.txt username@to_host:/remote/directory/
       scp -r local/directory/ username@to_host:/remote/directory/


    scp references: haydenjames.io

  2. Windows: Again, this process is a bit more complicated.
    1. A PuTTY + Command Prompt provides a version that is very similar to Mac/Linux scp (reference: tanaza), and requires minor changes in the call  (pscp -scp instead of scp).
    2. A Powershell specific command is a bit more complicated / not similar to Mac/Linux scp (reference: nsoftware)

 

Command line sftp

[to do: add section on sftp]

GUI “drag and drop” file sharing

A lot of students use GUI interfaces to transfer files instead of scp command line. These tools allow you to “drag and drop” files from one location to another after the connection is set up.

Some examples include:

  1. Mac:
    1. Cyberduck
  2. Linux:
    1. OpenSSH
  3. Windows: lots of options - thegeekstuff
    1. PuTTY
    2. WinSCP
    3. …


Basic CLI (Linux command line): bash

Understanding the basic Linux command line is important to be able to navigate when you are actually on the server. Below are some based commands to move around the file system, manipulate files, and view storage structure.

command translation
pwd present working directory
cd _dir_ change directory
rm _file_ remove / delete file
mv _file_ _dir_ move file (can change name)
cp _file_ _dir_ copy file
ls _dir_ list files in directory

use man _command_ to get a listing / manual of the command.


Additional resources that capture more information on the command line and are still geared to statistical computing can be found here: UCB 2015 reproducible DS lecture notes.

TODO: add information about sed, chmod, and a link to bash files.

Text editors

There are multiple options to view code on the server side. They include:

  • vim
  • … [to be added later]

Running jobs

We have included thoughts about running jobs in two places.

  1. The first looks at tools that allow you to keep running jobs on the server when you close the connection to the server (turn off your laptop, etc.). These tools are call terminal multiplexers. 
  2. The second set of tools focus on how to run your R analysis or Python analysis code on the server.

Server specs

Generally speaking, our servers have multiple CPU cores, but no GPU cores. For deep learning tasks that need GPUs, consider using the GPU cluster.

To examine the specs yourself, use 

cat /proc/meminfo
cat /proc/cpuinfo

to see some information.

Using the server respectfully

Because the servers are a shared department resource it is important to use them respectfully. There are many servers and multiple cores (CPUs) per server. To be respectful of other users we recommend checking (1) the current usage of any server you’re on (it’s possible one server is being used a lot but others are not), (2) not run a parallelize process that could use all of a servers cores (at least leave 1 core open), and (3) deprioritize your jobs that you don’t need to have done soon (especially if others are using the server when you submit your job).

  1. Understanding current resource usage:
    top lets you know the current usage of the server’s resources with a list of jobs run. It also tells you the percentage of the cores that are being used and the amount of RAM (memory) that is being used/ what is available. See howtogeek resources on how to interpret it all (the documentation is very clean, so we encourage you to read up there). 
    top
  2. Parallelizing with less than 100% of the cores:
    To avoid using all the cores when parallelizing, we recommend you first look at top, and then at least do (# of total cores) -1 as the upper limit of the number of cores used (less if others are using it as well).
  3. Deprioritizing when possible:
    nice/renice allows you to lower the priority of your jobs (so that others may use the resources if necessary) See howtogeek resources on the basics. nice sets the priority, and renice updates your priority. On the department servers, your can only deprioritize your jobs, everyone starts out on the same level. 
    nice --20 ./andrew_id
renice
Retrieved from "https://info.stat.cmu.edu/index.php?title=Linux_and_department_servers&oldid=179"