Stat DS Computing Wiki - User contributions [en]

Linux and department servers

2025-03-24T13:27:39Z

Ebensen: /* What resources exist */

The department has a set of servers, the hydras, that can help you run large analyses and gain a better understanding of common command line structure for more high scale computing. One can also view the department servers a bit more as a “sandbox” or “playground” as there are very few constraints and it’s okay to not run as polished code as you might on the [[PSC]] or Azure.

['''Request:''' Windows users (and all users) - if you see comments that are outdated / less helpful than expected please contribute.]

== What resources exist ==

# The department provides 13 servers (named {<code>hydra1</code>, ... , <code>hydra13</code>}<code>.stat.cmu.edu</code>.
# Each hydra has 12 CPU cores, and 64 GB of RAM (per core).
## The exceptions are hydra7 and hydra8, which have 128 GB of RAM. Use these if you have particularly large jobs that need lots of memory.
# The hydras all share the same data file system. This allows you to upload a file to any of the hydras and then have it on all of them.
# [[OpenMPI]] is installed
#* This allows MPI based parallelization to be used
#* Note that the hydras ARE NOT network connected, so MPI can only be used within a single hydra. It cannot communicate across multiple hydras.
#* If you need to run something in parallel across multiple nodes, you should use [[PSC]] instead.

''See [[what_resources_when| What resources when]] for guides to help decide which resource you should be using.''

== Using the servers respectfully ==
Because the servers are a shared department resource it is important to use them respectfully. There are many servers and multiple cores (CPUs) per server. To be respectful of other users:

# '''Check the current usage''' of the server you're on before using it.
# Do not use '''all''' the cores on a server, and do not use '''all''' the servers at the same time, so others can get work done.
# If you have a long-running job, '''deprioritize it''' so others can run too.

Each of these is easy to check.

=== Checking current usage ===
To check which servers are currently being used and which are open, see the [https://info.stat.cmu.edu/grafana/d/fdkb6l4klj400d/hydra-cluster?orgId=1 Grafana dashboard for the hydras]. It lists them all and their current usage.

When you're logged in to one node, you can use the <code>top</code> command. <code>top</code> displays the current usage of the server’s resources with a list of jobs run. It also tells you the percentage of the cores that are being used and the amount of RAM (memory) that is available. See [https://www.howtogeek.com/668986/how-to-use-the-linux-top-command-and-understand-its-output/ howtogeek] resources on how to interpret it all (the documentation is very clean, so we encourage you to read up ther

=== Limiting the cores you use ===
We recommend that when using parallel processing, use fewer than the total number of cores on the server. Most parallel processing packages let you choose the number of cores to use; the hydras have 12, so pick something smaller.

=== Deprioritize when possible ===
<code>nice/renice</code> allows you to lower the priority of your jobs, so the operating system lets other processes run first. See [https://www.howtogeek.com/411979/how-to-set-process-priorities-with-the-nice-and-renice-commands-in-linux/ howtogeek] resources on the basics. <code>nice</code> sets the priority, and <code>renice</code> updates your priority. On the department servers, everyone starts at the same priority, and you can only '''deprioritize''' your jobs -- you can't make them higher priority than everyone else.<syntaxhighlight lang="bash">nice -20 Rscript some-big-script.R
renice -n 20 processid # get the process ID from top</syntaxhighlight>
== Introduction to connecting to the server and useful Linux commands ==

The department servers all run Linux, and require you to use a command line to interact with it (as compared to Rstudio and other GUIs).
=== Getting Access: (<code>ssh</code>) ===

To get onto the server and type code you’ll need to access it through <code>ssh</code>.

<ul>
<li>For Macs (and Linux), open up a <code>terminal</code> and type
<syntaxhighlight lang="bash">ssh andrew_id@hydra1.stat.cmu.edu</syntaxhighlight>

you will then have to type in your password (if you don’t know it or don’t know it or have never accessed the server before - email [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu]).
</li>
<li>
For Windows, there's a few options (with different pros and cons). [https://www.makeuseof.com/tag/4-easy-ways-to-use-ssh-in-windows/ Makeuseof] provides an overview and introduction to how to get started on a few approaches, including PuTTY and Windows PowerShell.

Again, make sure to email [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu] to check that you have an account / or what your password is if you forgot.
</li>
</ul>
If you’re accessing the servers a lot, you might also look into aliases and password storing for the <code>ssh</code> commands. - Mac/Linux: [https://scotch.io/tutorials/how-to-create-an-ssh-shortcut scotch.io] - Windows: Powershell: [https://docs.microsoft.com/en-us/windows/terminal/tutorials/ssh microsoft], PuTTY: [https://www.makeuseof.com/tag/4-easy-ways-to-use-ssh-in-windows/ makeuseof]

=== File sharing (<code>scp</code> +) ===

There is also command line code to copy files from your personal computer to the server. This is done with a SCP (secure copy) tool. There are also GUIs (graphical user interfaces) that allow you to “drag and drop” files from your computer to the server (and visa-versa). Currently, a lot of students use GUIs to deal with file sharing.

==== Command line <code>scp</code>====

You can file share either by using <code>scp</code>, for example

<ol>
<li> Mac/Linux: This command is very similar to <code>cp</code> (the file copier).
<ol>
<li>Copy from server
<syntaxhighlight lang="bash"> scp username@from_host:path/to/file.txt /local/directory/ </syntaxhighlight>
<syntaxhighlight lang="bash">scp -r username@from_host:path/to/folder/ local/directory/</syntaxhighlight>
</li>
<li>Copy to server
 
<syntaxhighlight lang="bash"> scp local/file.txt username@to_host:/remote/directory/</syntaxhighlight>
<syntaxhighlight lang="bash"> scp -r local/directory/ username@to_host:/remote/directory/</syntaxhighlight>
</li>
</ol>

<code>scp</code> references: [https://haydenjames.io/linux-securely-copy-files-using-scp/ haydenjames.io]

</li>
<li>Windows: Again, this process is a bit more complicated.
<ol>
<li>A PuTTY + Command Prompt provides a version that is very similar to Mac/Linux <code>scp</code> (reference: [https://success.tanaza.com/s/article/How-to-use-SCP-command-on-Windows tanaza]), and requires minor changes in the call  (<code>pscp -scp</code> instead of <code>scp)</code>. </li>
<li>A Powershell specific command is a bit more complicated / not similar to Mac/Linux <code>scp</code> (reference: [https://www.nsoftware.com/kb/articles/psserver-scp.rst nsoftware]) </li>
</ol>
</li>
</ol>

===  GUI “drag-and-drop” file sharing ===
A '''lot''' of students use '''GUI''' interfaces to transfer files instead of <code>scp</code> command line. These tools allow you to “drag and drop” files from one location to another after the connection is set up.

Some examples include:

<ol>
<li>Mac:
<ol><li>
[https://cyberduck.io/ Cyberduck]
</li></ol></li>
<li>Linux:
<ol><li>
[https://www.openssh.com/ OpenSSH]
</li></ol></li>
<li> Windows: lots of options - [https://www.thegeekstuff.com/2011/06/windows-sftp-scp-clients/ thegeekstuff]
<ol>
<li>[https://www.putty.org/ PuTTY] </li>
<li>[https://winscp.net/eng/download.php WinSCP] </li></ol>
</li>
</ol>
=== Basic CLI (Linux command line): <code>bash</code>===

Understanding the basic Linux command line is important to be able to navigate when you are actually on the server. Below are some based commands to move around the file system, manipulate files, and view storage structure.

{| class="wikitable"
|-
! command
! translation
|-
| <code>pwd</code>
| print the current working directory
|-
| <code>cd _dir_</code>
| change directory
|-
| <code>rm _file_</code>
| remove / delete file
|-
| <code>mv _file_ _dir_</code>
| move file (can change name)
|-
| <code>cp _file_ _dir_</code>
| copy file
|-
| <code>ls _dir_</code>
| list files in directory
|}

use <code>man _command_</code> to get a listing / manual of the command.

Additional resources that capture more information on the command line and are still geared to statistical computing can be found here: [http://www.jarrodmillman.com/rcsds/standard/bash.html UCB 2015 reproducible DS lecture notes].

'''TODO: add information about <code>sed</code>, <code>chmod</code>, and a link to bash files.'''

== Text editors ==

There are multiple options to view code on the server side. They include:

*vim
*… [to be added later]

== Running jobs ==

We have included thoughts about running jobs in two places.

#The first looks at tools that allow you to keep running jobs on the server when you close the connection to the server (turn off your laptop, etc.). These tools are call [[Terminal_multiplexers|terminal multiplexers]]. 
#The second set of tools focus on how to run your [[R_on_the_server|R analysis]] or [[Python_on_the_server|Python analysis]] code on the server.

== Server specs ==

Generally speaking, our servers have multiple CPU cores, but no GPU cores. For deep learning tasks that need GPUs, consider using the [[GPU Cluster|GPU cluster]].

To examine the specs yourself, use
<syntaxhighlight lang="bash">cat /proc/meminfo
cat /proc/cpuinfo</syntaxhighlight>

to see some information.

Linux and department servers

2025-03-23T21:17:00Z

Ebensen:

The department has a set of servers, the hydras, that can help you run large analyses and gain a better understanding of common command line structure for more high scale computing. One can also view the department servers a bit more as a “sandbox” or “playground” as there are very few constraints and it’s okay to not run as polished code as you might on the [[PSC]] or Azure.

['''Request:''' Windows users (and all users) - if you see comments that are outdated / less helpful than expected please contribute.]

== What resources exist ==

# The department provides 13 servers (named {<code>hydra1</code>, ... , <code>hydra13</code>}<code>.stat.cmu.edu</code>.
# Each hydra has 12 CPU cores, and 64 GB of RAM (per core).
## The exceptions are hydra7 and hydra8, which have 128 GB of RAM. Use these if you have particularly large jobs that need lots of memory.
# The hydras all share the same data file system. This allows you to upload a file to any of the hydras and then have it on all of them.
# [[OpenMPI]] is installed
#* This allows MPI based parallelization to be used
#* Note that the hydras ARE NOT network connected, so MPI can only be used within a single hydra. It cannot communicate across multiple hydras.

''See [[what_resources_when| What resources when]] for guides to help decide which resource you should be using.''

== Using the servers respectfully ==
Because the servers are a shared department resource it is important to use them respectfully. There are many servers and multiple cores (CPUs) per server. To be respectful of other users:

# '''Check the current usage''' of the server you're on before using it.
# Do not use '''all''' the cores on a server, and do not use '''all''' the servers at the same time, so others can get work done.
# If you have a long-running job, '''deprioritize it''' so others can run too.

Each of these is easy to check.

=== Checking current usage ===
To check which servers are currently being used and which are open, see the [https://info.stat.cmu.edu/grafana/d/fdkb6l4klj400d/hydra-cluster?orgId=1 Grafana dashboard for the hydras]. It lists them all and their current usage.

When you're logged in to one node, you can use the <code>top</code> command. <code>top</code> displays the current usage of the server’s resources with a list of jobs run. It also tells you the percentage of the cores that are being used and the amount of RAM (memory) that is available. See [https://www.howtogeek.com/668986/how-to-use-the-linux-top-command-and-understand-its-output/ howtogeek] resources on how to interpret it all (the documentation is very clean, so we encourage you to read up ther

=== Limiting the cores you use ===
We recommend that when using parallel processing, use fewer than the total number of cores on the server. Most parallel processing packages let you choose the number of cores to use; the hydras have 12, so pick something smaller.

=== Deprioritize when possible ===
<code>nice/renice</code> allows you to lower the priority of your jobs, so the operating system lets other processes run first. See [https://www.howtogeek.com/411979/how-to-set-process-priorities-with-the-nice-and-renice-commands-in-linux/ howtogeek] resources on the basics. <code>nice</code> sets the priority, and <code>renice</code> updates your priority. On the department servers, everyone starts at the same priority, and you can only '''deprioritize''' your jobs -- you can't make them higher priority than everyone else.<syntaxhighlight lang="bash">nice -20 Rscript some-big-script.R
renice -n 20 processid # get the process ID from top</syntaxhighlight>
== Introduction to connecting to the server and useful Linux commands ==

The department servers all run Linux, and require you to use a command line to interact with it (as compared to Rstudio and other GUIs).
=== Getting Access: (<code>ssh</code>) ===

To get onto the server and type code you’ll need to access it through <code>ssh</code>.

<ul>
<li>For Macs (and Linux), open up a <code>terminal</code> and type
<syntaxhighlight lang="bash">ssh andrew_id@hydra1.stat.cmu.edu</syntaxhighlight>

you will then have to type in your password (if you don’t know it or don’t know it or have never accessed the server before - email [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu]).
</li>
<li>
For Windows, there's a few options (with different pros and cons). [https://www.makeuseof.com/tag/4-easy-ways-to-use-ssh-in-windows/ Makeuseof] provides an overview and introduction to how to get started on a few approaches, including PuTTY and Windows PowerShell.

Again, make sure to email [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu] to check that you have an account / or what your password is if you forgot.
</li>
</ul>
If you’re accessing the servers a lot, you might also look into aliases and password storing for the <code>ssh</code> commands. - Mac/Linux: [https://scotch.io/tutorials/how-to-create-an-ssh-shortcut scotch.io] - Windows: Powershell: [https://docs.microsoft.com/en-us/windows/terminal/tutorials/ssh microsoft], PuTTY: [https://www.makeuseof.com/tag/4-easy-ways-to-use-ssh-in-windows/ makeuseof]

=== File sharing (<code>scp</code> +) ===

There is also command line code to copy files from your personal computer to the server. This is done with a SCP (secure copy) tool. There are also GUIs (graphical user interfaces) that allow you to “drag and drop” files from your computer to the server (and visa-versa). Currently, a lot of students use GUIs to deal with file sharing.

==== Command line <code>scp</code>====

You can file share either by using <code>scp</code>, for example

<ol>
<li> Mac/Linux: This command is very similar to <code>cp</code> (the file copier).
<ol>
<li>Copy from server
<syntaxhighlight lang="bash"> scp username@from_host:path/to/file.txt /local/directory/ </syntaxhighlight>
<syntaxhighlight lang="bash">scp -r username@from_host:path/to/folder/ local/directory/</syntaxhighlight>
</li>
<li>Copy to server
 
<syntaxhighlight lang="bash"> scp local/file.txt username@to_host:/remote/directory/</syntaxhighlight>
<syntaxhighlight lang="bash"> scp -r local/directory/ username@to_host:/remote/directory/</syntaxhighlight>
</li>
</ol>

<code>scp</code> references: [https://haydenjames.io/linux-securely-copy-files-using-scp/ haydenjames.io]

</li>
<li>Windows: Again, this process is a bit more complicated.
<ol>
<li>A PuTTY + Command Prompt provides a version that is very similar to Mac/Linux <code>scp</code> (reference: [https://success.tanaza.com/s/article/How-to-use-SCP-command-on-Windows tanaza]), and requires minor changes in the call  (<code>pscp -scp</code> instead of <code>scp)</code>. </li>
<li>A Powershell specific command is a bit more complicated / not similar to Mac/Linux <code>scp</code> (reference: [https://www.nsoftware.com/kb/articles/psserver-scp.rst nsoftware]) </li>
</ol>
</li>
</ol>

===  GUI “drag-and-drop” file sharing ===
A '''lot''' of students use '''GUI''' interfaces to transfer files instead of <code>scp</code> command line. These tools allow you to “drag and drop” files from one location to another after the connection is set up.

Some examples include:

<ol>
<li>Mac:
<ol><li>
[https://cyberduck.io/ Cyberduck]
</li></ol></li>
<li>Linux:
<ol><li>
[https://www.openssh.com/ OpenSSH]
</li></ol></li>
<li> Windows: lots of options - [https://www.thegeekstuff.com/2011/06/windows-sftp-scp-clients/ thegeekstuff]
<ol>
<li>[https://www.putty.org/ PuTTY] </li>
<li>[https://winscp.net/eng/download.php WinSCP] </li></ol>
</li>
</ol>
=== Basic CLI (Linux command line): <code>bash</code>===

Understanding the basic Linux command line is important to be able to navigate when you are actually on the server. Below are some based commands to move around the file system, manipulate files, and view storage structure.

{| class="wikitable"
|-
! command
! translation
|-
| <code>pwd</code>
| print the current working directory
|-
| <code>cd _dir_</code>
| change directory
|-
| <code>rm _file_</code>
| remove / delete file
|-
| <code>mv _file_ _dir_</code>
| move file (can change name)
|-
| <code>cp _file_ _dir_</code>
| copy file
|-
| <code>ls _dir_</code>
| list files in directory
|}

use <code>man _command_</code> to get a listing / manual of the command.

Additional resources that capture more information on the command line and are still geared to statistical computing can be found here: [http://www.jarrodmillman.com/rcsds/standard/bash.html UCB 2015 reproducible DS lecture notes].

'''TODO: add information about <code>sed</code>, <code>chmod</code>, and a link to bash files.'''

== Text editors ==

There are multiple options to view code on the server side. They include:

*vim
*… [to be added later]

== Running jobs ==

We have included thoughts about running jobs in two places.

#The first looks at tools that allow you to keep running jobs on the server when you close the connection to the server (turn off your laptop, etc.). These tools are call [[Terminal_multiplexers|terminal multiplexers]]. 
#The second set of tools focus on how to run your [[R_on_the_server|R analysis]] or [[Python_on_the_server|Python analysis]] code on the server.

== Server specs ==

Generally speaking, our servers have multiple CPU cores, but no GPU cores. For deep learning tasks that need GPUs, consider using the [[GPU Cluster|GPU cluster]].

To examine the specs yourself, use
<syntaxhighlight lang="bash">cat /proc/meminfo
cat /proc/cpuinfo</syntaxhighlight>

to see some information.

Linux and department servers

2025-03-23T21:15:20Z

Ebensen: Edited OpenMPI comment on the hydra page to reflect the fact that the hydras are not network connected.

The department has a set of servers, the hydras, that can help you run large analyses and gain a better understanding of common command line structure for more high scale computing. One can also view the department servers a bit more as a “sandbox” or “playground” as there are very few constraints and it’s okay to not run as polished code as you might on the [[PSC]] or Azure.

['''Request:''' Windows users (and all users) - if you see comments that are outdated / less helpful than expected please contribute.]

== What resources exist ==

# The department provides 13 servers (named {<code>hydra1</code>, ... , <code>hydra13</code>}<code>.stat.cmu.edu</code>.
# Each hydra has 12 CPU cores, and 64 GB of RAM (per core).
## The exceptions are hydra7 and hydra8, which have 128 GB of RAM. Use these if you have particularly large jobs that need lots of memory.
# The hydras all share the same data file system. This allows you to upload a file to any of the hydras and then have it on all of them.
# [[OpenMPI]] is installed
#* This allows MPI based parallelization to be used
#* Note that the hydras ARE NOT network connected, so MPI can only be used within a single hydra. It cannot communicate across hydras.

''See [[what_resources_when| What resources when]] for guides to help decide which resource you should be using.''

== Using the servers respectfully ==
Because the servers are a shared department resource it is important to use them respectfully. There are many servers and multiple cores (CPUs) per server. To be respectful of other users:

# '''Check the current usage''' of the server you're on before using it.
# Do not use '''all''' the cores on a server, and do not use '''all''' the servers at the same time, so others can get work done.
# If you have a long-running job, '''deprioritize it''' so others can run too.

Each of these is easy to check.

=== Checking current usage ===
To check which servers are currently being used and which are open, see the [https://info.stat.cmu.edu/grafana/d/fdkb6l4klj400d/hydra-cluster?orgId=1 Grafana dashboard for the hydras]. It lists them all and their current usage.

When you're logged in to one node, you can use the <code>top</code> command. <code>top</code> displays the current usage of the server’s resources with a list of jobs run. It also tells you the percentage of the cores that are being used and the amount of RAM (memory) that is available. See [https://www.howtogeek.com/668986/how-to-use-the-linux-top-command-and-understand-its-output/ howtogeek] resources on how to interpret it all (the documentation is very clean, so we encourage you to read up ther

=== Limiting the cores you use ===
We recommend that when using parallel processing, use fewer than the total number of cores on the server. Most parallel processing packages let you choose the number of cores to use; the hydras have 12, so pick something smaller.

=== Deprioritize when possible ===
<code>nice/renice</code> allows you to lower the priority of your jobs, so the operating system lets other processes run first. See [https://www.howtogeek.com/411979/how-to-set-process-priorities-with-the-nice-and-renice-commands-in-linux/ howtogeek] resources on the basics. <code>nice</code> sets the priority, and <code>renice</code> updates your priority. On the department servers, everyone starts at the same priority, and you can only '''deprioritize''' your jobs -- you can't make them higher priority than everyone else.<syntaxhighlight lang="bash">nice -20 Rscript some-big-script.R
renice -n 20 processid # get the process ID from top</syntaxhighlight>
== Introduction to connecting to the server and useful Linux commands ==

The department servers all run Linux, and require you to use a command line to interact with it (as compared to Rstudio and other GUIs).
=== Getting Access: (<code>ssh</code>) ===

To get onto the server and type code you’ll need to access it through <code>ssh</code>.

<ul>
<li>For Macs (and Linux), open up a <code>terminal</code> and type
<syntaxhighlight lang="bash">ssh andrew_id@hydra1.stat.cmu.edu</syntaxhighlight>

you will then have to type in your password (if you don’t know it or don’t know it or have never accessed the server before - email [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu]).
</li>
<li>
For Windows, there's a few options (with different pros and cons). [https://www.makeuseof.com/tag/4-easy-ways-to-use-ssh-in-windows/ Makeuseof] provides an overview and introduction to how to get started on a few approaches, including PuTTY and Windows PowerShell.

Again, make sure to email [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu] to check that you have an account / or what your password is if you forgot.
</li>
</ul>
If you’re accessing the servers a lot, you might also look into aliases and password storing for the <code>ssh</code> commands. - Mac/Linux: [https://scotch.io/tutorials/how-to-create-an-ssh-shortcut scotch.io] - Windows: Powershell: [https://docs.microsoft.com/en-us/windows/terminal/tutorials/ssh microsoft], PuTTY: [https://www.makeuseof.com/tag/4-easy-ways-to-use-ssh-in-windows/ makeuseof]

=== File sharing (<code>scp</code> +) ===

There is also command line code to copy files from your personal computer to the server. This is done with a SCP (secure copy) tool. There are also GUIs (graphical user interfaces) that allow you to “drag and drop” files from your computer to the server (and visa-versa). Currently, a lot of students use GUIs to deal with file sharing.

==== Command line <code>scp</code>====

You can file share either by using <code>scp</code>, for example

<ol>
<li> Mac/Linux: This command is very similar to <code>cp</code> (the file copier).
<ol>
<li>Copy from server
<syntaxhighlight lang="bash"> scp username@from_host:path/to/file.txt /local/directory/ </syntaxhighlight>
<syntaxhighlight lang="bash">scp -r username@from_host:path/to/folder/ local/directory/</syntaxhighlight>
</li>
<li>Copy to server
 
<syntaxhighlight lang="bash"> scp local/file.txt username@to_host:/remote/directory/</syntaxhighlight>
<syntaxhighlight lang="bash"> scp -r local/directory/ username@to_host:/remote/directory/</syntaxhighlight>
</li>
</ol>

<code>scp</code> references: [https://haydenjames.io/linux-securely-copy-files-using-scp/ haydenjames.io]

</li>
<li>Windows: Again, this process is a bit more complicated.
<ol>
<li>A PuTTY + Command Prompt provides a version that is very similar to Mac/Linux <code>scp</code> (reference: [https://success.tanaza.com/s/article/How-to-use-SCP-command-on-Windows tanaza]), and requires minor changes in the call  (<code>pscp -scp</code> instead of <code>scp)</code>. </li>
<li>A Powershell specific command is a bit more complicated / not similar to Mac/Linux <code>scp</code> (reference: [https://www.nsoftware.com/kb/articles/psserver-scp.rst nsoftware]) </li>
</ol>
</li>
</ol>

===  GUI “drag-and-drop” file sharing ===
A '''lot''' of students use '''GUI''' interfaces to transfer files instead of <code>scp</code> command line. These tools allow you to “drag and drop” files from one location to another after the connection is set up.

Some examples include:

<ol>
<li>Mac:
<ol><li>
[https://cyberduck.io/ Cyberduck]
</li></ol></li>
<li>Linux:
<ol><li>
[https://www.openssh.com/ OpenSSH]
</li></ol></li>
<li> Windows: lots of options - [https://www.thegeekstuff.com/2011/06/windows-sftp-scp-clients/ thegeekstuff]
<ol>
<li>[https://www.putty.org/ PuTTY] </li>
<li>[https://winscp.net/eng/download.php WinSCP] </li></ol>
</li>
</ol>
=== Basic CLI (Linux command line): <code>bash</code>===

Understanding the basic Linux command line is important to be able to navigate when you are actually on the server. Below are some based commands to move around the file system, manipulate files, and view storage structure.

{| class="wikitable"
|-
! command
! translation
|-
| <code>pwd</code>
| print the current working directory
|-
| <code>cd _dir_</code>
| change directory
|-
| <code>rm _file_</code>
| remove / delete file
|-
| <code>mv _file_ _dir_</code>
| move file (can change name)
|-
| <code>cp _file_ _dir_</code>
| copy file
|-
| <code>ls _dir_</code>
| list files in directory
|}

use <code>man _command_</code> to get a listing / manual of the command.

Additional resources that capture more information on the command line and are still geared to statistical computing can be found here: [http://www.jarrodmillman.com/rcsds/standard/bash.html UCB 2015 reproducible DS lecture notes].

'''TODO: add information about <code>sed</code>, <code>chmod</code>, and a link to bash files.'''

== Text editors ==

There are multiple options to view code on the server side. They include:

*vim
*… [to be added later]

== Running jobs ==

We have included thoughts about running jobs in two places.

#The first looks at tools that allow you to keep running jobs on the server when you close the connection to the server (turn off your laptop, etc.). These tools are call [[Terminal_multiplexers|terminal multiplexers]]. 
#The second set of tools focus on how to run your [[R_on_the_server|R analysis]] or [[Python_on_the_server|Python analysis]] code on the server.

== Server specs ==

Generally speaking, our servers have multiple CPU cores, but no GPU cores. For deep learning tasks that need GPUs, consider using the [[GPU Cluster|GPU cluster]].

To examine the specs yourself, use
<syntaxhighlight lang="bash">cat /proc/meminfo
cat /proc/cpuinfo</syntaxhighlight>

to see some information.

VSCode

2024-09-13T19:47:44Z

Ebensen: /* Tunnelling */ edited export path instructions to be generic instead of using a path specific to PSC

VSCode is an open-source IDE maintained by Microsoft, and it is considered a “lightweight” code editor. In comparison with full integrated development environment (IDE) editors which occupy gigabytes of disk space, VS Code uses less than 200MB when installed. Despite the “lightweight” term, Visual Studio Code offers a massive number of features (via so-called “extensions”) which keep increasing and improving with every new update. 

Resources:

#[https://code.visualstudio.com/docs Official documentation]
#[https://code.visualstudio.com/download Download]

 

== Basic Usage ==

Some good extensions: 

#Python (and Pylance)
#Jupyter
#R
#LaTeX Workshop
#Remote - SSH
#autoDocstring

== Remote Development and Prototyping via SSH ==

In this section we will learn how to use VSCode to manage, develop, and test code that we have stored on a remote server (like [https://www.stat.cmu.edu/computewiki/index.php/Linux_and_department_servers Hydras], [https://www.stat.cmu.edu/computewiki/index.php/Azure_getting_started Azure], etc...).

This is particularly useful to avoid painful workflows that see us doing the following:

#develop locally on our laptop
#scp some scripts to the remote server and run them (assuming we have consistent environments)
#scp the results to the local laptop
#repeat from 1

Ideally, we want to be able to use the remote server only as the computational workhorse, while leveraging the GUI of our local operative system. Doing this in VSCode is extremely simple:

#In the left toolbar, click on the “Extensions” button and search for SSH. The correct one is hte one developed by Microsoft (blue icon). Click on the install button.

[[File:Ssh.png|1000x1000px|RTENOTITLE]]

 
<ol start="2">
<li>Once the extension is installed, click on the small green button at the very bottom-left of the window (the one with a symbol similar to ><). A pop-up window at the top will open: click on “Connect to Host”</li>
<li>At this point, we have two alternatives:
# add a new SSH host manually, which will prompt another small window where to enter the connection details (similar to what one does from the terminal)
#or '''better''', if we have a SSH <code>config </code>file in the <code>.ssh </code>directory, VSCode will automatically list the possible remote servers that are specified in this file. If we specified a path to a key, then VSCode will connect automatically. If instead the remote server requires a password (as it is the case for Hydras), then we will be prompted with another window to enter it. As an example:
[[File:Hosts.png|RTENOTITLE]]
 [[File:Ssh config.png|RTENOTITLE]]
<ol start="4">
<li>Congratulations! You are now connected to a remote server through VSCode. By clicking on the “Explorer” button at the top in the left-side toolbar, you can navigate the file system of the server, develop and run code, etc. Everything you do and see is actually being hosted, run, and saved remotely. 
[[File:Remote file.png|1000x1000px|RTENOTITLE]]</li>
</ol>
</li>
</ol>

 

== Managing environments within VSCode ==

Since each project might (and should) have its own development environment that you setup, with dependencies and packages that make it unique for the tasks of this project, VSCode allows to easily choose which interpreter to use. 

We simply have to click at the bottom right of the window, on the button that shows the current Python (or other language) version that we are using. This will open a prompt to choose the environment among those present on the machine we are developing on:

[[File:Env.png|1000x1000px|Env.png]]

 

== Support for Jupyter Notebooks (even remote!) within VSCode ==

In one of the previous section we learned how to setup VSCode for remote development. A great feature of the workflow that we have just enabled is that we can run Jupyter Notebooks (and produce plots, tables, etc...) that are hosted remotely while seeing the output on our local laptop within VSCode (this works also if we are running things locally). Doing so is very easy:

#Click on the “Extensions” button on the left toolbar and download the Jupyter extension

[[File:Jupyter.png|1000x1000px|Jupyter.png]]
<ol start="2">
<li>Open a .ipynb file and choose the kernel (interpreter for the correct project environment) from the button that appears at the top right of the opened notebook</li>
</ol>

[[File:Choose kernel.png|1000x1000px|Choose kernel.png]]
<ol start="3">
<li>Happy coding!</li>
</ol>

== Support for RStudio (even remote!) within VSCode ==

Install the R extension, which provides support for the [https://www.r-project.org/ R programming language], including features such as extended syntax highlighting, R language service based on code analysis, interacting with R terminals, viewing data, plots, workspace variables, help pages, managing packages, and working with [https://rmarkdown.rstudio.com/ R Markdown] documents.

== Miscellanea ==

=== Amphetamine ===

If you need your laptop to stay awake while there's a process running --- you are running some cell in a notebook via remote SSH --- check out this [[Amphetamine|article in the wiki]].

=== Tunnelling ===

Disclaimer: this is particularly useful for PSC (Pittsburgh Supercomuting Center) users, as it allows to connect directly to a compute node on PSC and have things like Jupyter Notebooks run via VSCode without problems. This solution bypasses a problem incurred when connecting via SSH from VSCode, which caused to land on the login nodes. These have very little memory and basically do not support any computation.

Instructions for creating a VS code tunnel to PSC:

*First set up code CLI on PSC: [https://dev.to/burkeholland/introducing-vs-code-remote-tunnels-connect-to-remote-machines-with-ease-3nlg https://dev.to/burkeholland/introducing-vs-code-remote-tunnels-connect-to-remote-machines-with-ease-3nlg ]
*Then run the following:
<pre> curl -Lk 'https://code.visualstudio.com/sha/download?build=stable&os=cli-alpine-x64' --output vscode_cli.tar.gz
tar -xf vscode_cli.tar.gz
</pre>

           This will create a "code" executable file which we need to add to the path. Open <code>.bashrc</code> (or <code>.zshrc </code>if you have a recent Mac) and edit the line that exports the path to be your current directory. If need be, you can get the path to your current directory using the command <code>pwd</code>
<pre>export PATH="<current directory>:$PATH"</pre>

*Restart the shell and check that "which code" finds the installation.
*Next, install the remote tunnels and remote ssh extensions on VS code.
**These are found in the extensions menu
**After installation there should be a blue "open remote window" (>< symbol) in the bottom left of the VS code window.

*Next, we set up the tunnel: [https://dev.to/burkeholland/introducing-vs-code-remote-tunnels-connect-to-remote-machines-with-ease-3nlg https://dev.to/burkeholland/introducing-vs-code-remote-tunnels-connect-to-remote-machines-with-ease-3nlg ]
**Start interactive job
**Run <code>code tunnel</code> and follow the instructions. I recommend logging in via GitHub the first time and occasionally you will be asked to name the machine. By default it will name it based off of the node you are currently interacting with; however, the tunnel will connect to any node at PSC via the same setup so I recommend you name it something generic related to PSC. You will eventually see a message like “Open this link in your browser https://vscode.dev/tunnel/<machine name>”
**On VS Code on your local machine, click the blue "open remote window" (>< symbol) in the bottom left and select Connect to Tunnel.
**Select GitHub and then the machine that matches <machine name> in the link shown on the remote location. This should reload your VS code window to be running on the remote server, but if you started an interactive job it is running on a compute node.
**To stop the connection, run "ctrl + c"
**If you need to deregister the tunnel run "code tunnel unregister" however, this should be unnecessary

For any future connection you run this same set of steps, as I haven't found a way to avoid re logging in via github every time.

VSCode

2024-09-11T19:23:41Z

Ebensen: Changed the tunnelling path description to mention it is only for Bridges 2

VSCode is an open-source IDE maintained by Microsoft, and it is considered a “lightweight” code editor. In comparison with full integrated development environment (IDE) editors which occupy gigabytes of disk space, VS Code uses less than 200MB when installed. Despite the “lightweight” term, Visual Studio Code offers a massive number of features (via so-called “extensions”) which keep increasing and improving with every new update. 

Resources:

#[https://code.visualstudio.com/docs Official documentation]
#[https://code.visualstudio.com/download Download]

 

== Basic Usage ==

Some good extensions: 

#Python (and Pylance)
#Jupyter
#R
#LaTeX Workshop
#Remote - SSH
#autoDocstring

== Remote Development and Prototyping via SSH ==

In this section we will learn how to use VSCode to manage, develop, and test code that we have stored on a remote server (like [https://www.stat.cmu.edu/computewiki/index.php/Linux_and_department_servers Hydras], [https://www.stat.cmu.edu/computewiki/index.php/Azure_getting_started Azure], etc...).

This is particularly useful to avoid painful workflows that see us doing the following:

#develop locally on our laptop
#scp some scripts to the remote server and run them (assuming we have consistent environments)
#scp the results to the local laptop
#repeat from 1

Ideally, we want to be able to use the remote server only as the computational workhorse, while leveraging the GUI of our local operative system. Doing this in VSCode is extremely simple:

#In the left toolbar, click on the “Extensions” button and search for SSH. The correct one is hte one developed by Microsoft (blue icon). Click on the install button.

[[File:Ssh.png|1000x1000px|RTENOTITLE]]

 
<ol start="2">
<li>Once the extension is installed, click on the small green button at the very bottom-left of the window (the one with a symbol similar to ><). A pop-up window at the top will open: click on “Connect to Host”</li>
<li>At this point, we have two alternatives:
# add a new SSH host manually, which will prompt another small window where to enter the connection details (similar to what one does from the terminal)
#or '''better''', if we have a SSH <code>config </code>file in the <code>.ssh </code>directory, VSCode will automatically list the possible remote servers that are specified in this file. If we specified a path to a key, then VSCode will connect automatically. If instead the remote server requires a password (as it is the case for Hydras), then we will be prompted with another window to enter it. As an example:
[[File:Hosts.png|RTENOTITLE]]
 [[File:Ssh config.png|RTENOTITLE]]
<ol start="4">
<li>Congratulations! You are now connected to a remote server through VSCode. By clicking on the “Explorer” button at the top in the left-side toolbar, you can navigate the file system of the server, develop and run code, etc. Everything you do and see is actually being hosted, run, and saved remotely. 
[[File:Remote file.png|1000x1000px|RTENOTITLE]]</li>
</ol>
</li>
</ol>

 

== Managing environments within VSCode ==

Since each project might (and should) have its own development environment that you setup, with dependencies and packages that make it unique for the tasks of this project, VSCode allows to easily choose which interpreter to use. 

We simply have to click at the bottom right of the window, on the button that shows the current Python (or other language) version that we are using. This will open a prompt to choose the environment among those present on the machine we are developing on:

[[File:Env.png|1000x1000px|Env.png]]

 

== Support for Jupyter Notebooks (even remote!) within VSCode ==

In one of the previous section we learned how to setup VSCode for remote development. A great feature of the workflow that we have just enabled is that we can run Jupyter Notebooks (and produce plots, tables, etc...) that are hosted remotely while seeing the output on our local laptop within VSCode (this works also if we are running things locally). Doing so is very easy:

#Click on the “Extensions” button on the left toolbar and download the Jupyter extension

[[File:Jupyter.png|1000x1000px|Jupyter.png]]
<ol start="2">
<li>Open a .ipynb file and choose the kernel (interpreter for the correct project environment) from the button that appears at the top right of the opened notebook</li>
</ol>

[[File:Choose kernel.png|1000x1000px|Choose kernel.png]]
<ol start="3">
<li>Happy coding!</li>
</ol>

== Support for RStudio (even remote!) within VSCode ==

Install the R extension, which provides support for the [https://www.r-project.org/ R programming language], including features such as extended syntax highlighting, R language service based on code analysis, interacting with R terminals, viewing data, plots, workspace variables, help pages, managing packages, and working with [https://rmarkdown.rstudio.com/ R Markdown] documents.

== Miscellanea ==

=== Amphetamine ===

If you need your laptop to stay awake while there's a process running --- you are running some cell in a notebook via remote SSH --- check out this [[Amphetamine|article in the wiki]].

=== Tunnelling ===

Disclaimer: this is particularly useful for PSC (Pittsburgh Supercomuting Center) users, as it allows to connect directly to a compute node on PSC and have things like Jupyter Notebooks run via VSCode without problems. This solution bypasses a problem incurred when connecting via SSH from VSCode, which caused to land on the login nodes. These have very little memory and basically do not support any computation.

Instructions for creating a VS code tunnel to PSC:

*First set up code CLI on PSC: [https://dev.to/burkeholland/introducing-vs-code-remote-tunnels-connect-to-remote-machines-with-ease-3nlg https://dev.to/burkeholland/introducing-vs-code-remote-tunnels-connect-to-remote-machines-with-ease-3nlg ]
*Then run the following:
<pre> curl -Lk 'https://code.visualstudio.com/sha/download?build=stable&os=cli-alpine-x64' --output vscode_cli.tar.gz
tar -xf vscode_cli.tar.gz
</pre>

           This will create a "code" executable file which we need to add to the path. Open <code>.bashrc</code> (or <code>.zshrc </code>if you have a recent Mac) and edit the line that exports the path to be your current directory. For example on the Bridges 2 supercomputer.
<pre>export PATH="/jet/home/<username>:$PATH"</pre>

*Restart the shell and check that "which code" finds the installation.
*Next, install the remote tunnels and remote ssh extensions on VS code.
**These are found in the extensions menu
**After installation there should be a blue "open remote window" (>< symbol) in the bottom left of the VS code window.

*Next, we set up the tunnel: [https://dev.to/burkeholland/introducing-vs-code-remote-tunnels-connect-to-remote-machines-with-ease-3nlg https://dev.to/burkeholland/introducing-vs-code-remote-tunnels-connect-to-remote-machines-with-ease-3nlg ]
**Start interactive job
**Run <code>code tunnel</code> and follow the instructions. I recommend logging in via GitHub the first time and occasionally you will be asked to name the machine. By default it will name it based off of the node you are currently interacting with; however, the tunnel will connect to any node at PSC via the same setup so I recommend you name it something generic related to PSC. You will eventually see a message like “Open this link in your browser https://vscode.dev/tunnel/<machine name>”
**On VS Code on your local machine, click the blue "open remote window" (>< symbol) in the bottom left and select Connect to Tunnel.
**Select GitHub and then the machine that matches <machine name> in the link shown on the remote location. This should reload your VS code window to be running on the remote server, but if you started an interactive job it is running on a compute node.
**To stop the connection, run "ctrl + c"
**If you need to deregister the tunnel run "code tunnel unregister" however, this should be unnecessary

For any future connection you run this same set of steps, as I haven't found a way to avoid re logging in via github every time.