Stat DS Computing Wiki - User contributions [en]

ACCESS

2025-09-03T03:22:53Z

Skipper: Replaced content with "This will shortly be updated to add easy access to ACCESS On-Ramps: <div id="resource-catalog-react"></div> <noscript>This page loads an interactive catalog that requires..."

This will shortly be updated to add easy access to ACCESS On-Ramps: <div id="resource-catalog-react"></div>
<noscript>This page loads an interactive catalog that requires JavaScript.</noscript>

MediaWiki:Common.js

2025-09-03T03:22:13Z

Skipper: Created page with "/* Any JavaScript here will be loaded for all users on every page load. */ // Load Bootstrap Icons CSS (site-wide) (function () { if (document.getElementById('bi-cdn-link'))..."

/* Any JavaScript here will be loaded for all users on every page load. */
// Load Bootstrap Icons CSS (site-wide)
(function () {
if (document.getElementById('bi-cdn-link')) return;
var link = document.createElement('link');
link.id = 'bi-cdn-link';
link.rel = 'stylesheet';
link.href = 'https://cdn.jsdelivr.net/npm/bootstrap-icons@1.11.3/font/bootstrap-icons.min.css';
document.head.appendChild(link);
})();

// ACCESS On-Ramps embed via ES module, mounted when placeholder div is present
(function () {
function mountOnRamps() {
var target = document.getElementById('resource-catalog-react');
if (!target) return; // Only act on pages that have the placeholder

// Build a <script type="module"> that mirrors ACCESS instructions
var script = document.createElement('script');
script.type = 'module';
script.textContent = `
const baseUrl = "https://esm.sh/@xras/ui@onramps_v1/dist";
import {
onRampsResourceCatalog,
shadowTarget,
} from "https://esm.sh/@xras/ui@onramps_v1/dist/xras-ui.js";

onRampsResourceCatalog({
target: shadowTarget(
document.getElementById("resource-catalog-react"),
{ accessStyles: true, baseUrl }
),
onRamps: true,
baseUrl
});
`;
document.body.appendChild(script);
}

// Run after the page is ready (and after MediaWiki’s JS pipelines)
if (document.readyState === 'loading') {
document.addEventListener('DOMContentLoaded', mountOnRamps);
} else {
mountOnRamps();
}
})();

ACCESS

2025-09-03T03:18:38Z

Skipper:

The NSF’s ACCESS (Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support) program builds upon the successes of the 11-year XSEDE project.

NSF funded supercomputers, like PSC are available via the ACCESS federated system.  Besides parallel or memory-intensive computing options, there are many GPU resources available to ACCESS users via the Credit Marketplace.

#If you are a Graduate Student, all you need is '''an account''',''' '''your''' CV''', a '''1-paragraph abstract''' and '''a letter from your advisor'''.  (Carl has a template letter your advisor can sign.) With these 4 items, you can complete a simple form and get up to 400,000 CPU-hours to exchange for access to most of the nation's supercomputing cyberinfrastructure. Check out the following [https://allocations.access-ci.org/preparing-your-explore-access-request link].
#If you previously held an XSEDE account, you can login here: [https://identity.access-ci.org/new-user-federated https://identity.access-ci.org/new-user-federated] and complete your account transition from XSEDE to ACCESS by selecting “ACCESS CI (XSEDE)” and subsequently login using Andrew Web Login.  You will then be guided through a short verification process similar to any other website. 
#If you are '''new''' to the department or supercomputing in general, I set up a new account so you will be able to access (no pun intended) these resources easily now or in the future.  Visit [https://identity.access-ci.org/new-user-federated https://identity.access-ci.org/new-user-federated] to set up a new account -  make sure to choose “'''Carnegie Mellon University'''”! 
#The interface to the Supercomputing Credit Market is still being improved, but we are working on documentation to address that.

This will shortly be updated to add easy access to ACCESS On-Ramps: <div id="resource-catalog-react"></div>
<noscript>This page loads an interactive catalog that requires JavaScript.</noscript>

Parallelization in python

2025-05-16T14:46:25Z

Skipper: Replaced incorrectly migrated page with correct content.

'''This article is still being developed - released earlier to align with Department seminar.'''

In python, parallelization is a bit more of a “wild west” in terms of the number of packages that can help you parallelize your processes (for a long . This wiki will attempt to present minor examples of some along with an outline of things to think about in selecting the library you end up using. In general, the process in python involves

# defining a function that captures what you want to have parallelized
# call that function repeatedly in the the parallelization package way (which often looks like a list-recursive call).

There are a few different general frameworks to accomplish this.

# '''general wrappers''', define the cores you’ll be using and then wrap that around the function call in a list-recursive way
# '''just in time compilers''', in addition to providing parallelization tools these packages provide decorators (e.g. <code>@jit</code>) for functions that allow for just in time compilation of the python function into lower level machine code (like <code>C++</code>)

We will discuss a python package that is build specifically to parallelize jobs inside '''jupyter notebooks'''.

= Non-parallelized: single core example =

We naturally want to compare a parallelization approach with the base single core analysis, which can be done with the code as follows:

<syntaxhighlight lang="python">#base for-loop
import numpy as np
from sklearn.linear_model import LinearRegression

my_k = list(np.arange(14)+1)*20

def my_simulation(k):
"""
estamine the polynomial fit of degree k's MSE for a linear
relationship

Parameters
----------
k : int
degree of polynomial fit

Returns
-------
mse of fit based on 30 points applied to a test set of 10000 points
"""
x = np.random.uniform(size=30,low=-3,high=3)
y = 2*x + 2* np.random.normal(size=30)
y = y.reshape(-1,1)
X_poly = np.concatenate([x.reshape(-1,1)**kk
for kk in np.arange(1,k+1,dtype = int)],
axis = 1)

x0 = np.random.uniform(size=10000,low=-3,high=3)
y0 = 2*x0 + 2*np.random.normal(size=10000)
y0 = y0.reshape(-1,1)
X0_poly = np.concatenate([x0.reshape(-1,1)**kk
for kk in np.arange(1,k+1,dtype = int)],
axis = 1)

my_lm = LinearRegression().fit(X_poly,y)
y0_hat = my_lm.predict(X0_poly)

return(np.mean((y0-y0_hat)**2))
</syntaxhighlight>
<syntaxhighlight lang="python">%%timeit -n3

[my_simulation(k) for k in my_k]</syntaxhighlight>
<syntaxhighlight lang="python">676 ms ± 37.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)</syntaxhighlight>
= General python parallelization packages =

As mentioned there are a lot of package to do general parallelization. Two commonly used ones are <code>joblib</code> and <code>multiprocessing</code>. <code>joblib</code> looks a bit more like list-recursion and <code>multiprocessing</code> uses functions like <code>.map()</code>. <code>multiprocessing</code> can have problems with <code>ipython</code> and <code>jupyter notebook</code>s (though <code>joblib</code> wraps it and <code>multiprocess</code> is just a different fork of <code>multiprocessing</code>).

== <code>multiprocessing</code>/<code>multiprocess</code> ==

The <code>multiprocessing</code> package is actually built into python, and works in a pretty straightforward manner. It also allows for synchronous vs asynchronous parallelization, where synchronous means the order of the jobs must be followed and asynchronous where the order of jobs need not be followed. Many statistical uses of parallelization does not need synchronous parallelization.

There since both approaches have the same syntax we’ll be using <code>multiprocess</code> (as it works with <code>ipython</code> and <code>jupyter notebooks</code>).

=== parallelization ===

First, you need to identify the number of cores you have:

<syntaxhighlight lang="python">import multiprocess as mp
#import multiprocessing as mp

num_cores = mp.cpu_count()</syntaxhighlight>
==== synchronous ====

<syntaxhighlight lang="python">%%timeit

with mp.Pool(processes=num_cores-2) as p:
results = p.map(my_simulation, my_k)</syntaxhighlight>
<syntaxhighlight lang="python">209 ms ± 24.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)</syntaxhighlight>
==== asynchronous ====

<syntaxhighlight lang="python">%%timeit

with mp.Pool(processes=num_cores-2) as p:
results = p.map_async(my_simulation, my_k).get() # not clear why I needed to use .get() to get something returnable for async</syntaxhighlight>
<syntaxhighlight lang="python">48.9 ms ± 3.08 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)</syntaxhighlight>
== <code>joblib</code> ==

The <code>joblib</code> package is pretty straightforward and requires you to structure things in terms of list-recursion with a few wrappers to help the process work as desired.

=== setup ===

<syntaxhighlight lang="python">import joblib

num_cores = joblib.cpu_count()</syntaxhighlight>
=== implimentation ===

To use <code>joblib</code> functionality we can import the functions <code>Parallel</code> and <code>delayed</code>.

<syntaxhighlight lang="python">from joblib import Parallel, delayed</syntaxhighlight>
We then can move our list-recursive example into our <code>joblib</code> structure. We wrap <code>Parallel(...)()</code> around the old of the inside of the list-recursion and <code>delayed()</code> around the functions that will be repeated multiple times.

<syntaxhighlight lang="python">%% timeit

r = Parallel(n_jobs=num_cores-2)(delayed(my_simulation)(k) for k in my_k)</syntaxhighlight>
<syntaxhighlight lang="python">172 ms ± 23.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)</syntaxhighlight>
=== Add-ons ===

There are multiple add-ons that can be desirable

<ol style="list-style-type: decimal;">
<li>'''verbosity''' can be added with a single parameter use (as seen below) - the large the value, the more messages:
<syntaxhighlight lang="python">r = Parallel(n_jobs=num_cores-2, verbose=10)(delayed(my_simulation)(k) for k in my_k)</syntaxhighlight></li>
<li>'''multiple parameters''' can be passed with redefinitions of functions to take groupings. To replace multiple for-loops often the function <code>itertools.product()</code> is used for the parameters.</li></ol>

= Just in Time compilers + parallelization =

TODO: <code>numba</code> and <code>cython</code>. The idea of these is to compile the function you will be using the first time you use it and then never compile it again (speeding up the actual use of the function). That is then combined with standard parallelization

= Jupyter notebook: <code>ipyparallel</code> =

For those that heavily use jupyter notebooks and nothing else (which this author personally doesn’t encourage), the package <code>ipyparallel</code> nicely works inside jupyter notebooks (but so does many of the other packages above like <code>joblib</code>, <code>multiprocess</code>, etc.). One nice this about this package is the verbosity, etc. works intellegently in the notebook. This package also works with <code>numba.jit</code> and <code>cython</code>.

Before opening your <code>jupyter notebook</code> you’ll want to make sure the extension is enabled - which can be down from the command line with the following command or inside the <code>jupyter notebook</code> extensions tab.

<syntaxhighlight lang="bash">ipcluster nbextension enable</syntaxhighlight>
And then you’ll need to go to the “IPython Clusters” tab in <code>jupyter noteook</code> and start some parallel clusters.

== <code>ipyparallel</code> parallelization ==

<syntaxhighlight lang="python">import ipyparallel as ipp</syntaxhighlight>
For direct execution we define <code>dv</code> and provide it with what it needs…

<syntaxhighlight lang="python">rc = ipp.Client()
dv = rc[:] # direct view</syntaxhighlight>
=== passing things to each cluster ===

<code>ipyparallel</code> provides ways for the user to pass only the necessary items to the the clusters (as to not overload the clusters with too many things).

<ol style="list-style-type: decimal;">
<li>passing packages all together
<syntaxhighlight lang="python">with dv.sync_imports():
from sklearn.linear_model import LinearRegression
# getting dependencies on the clusters</syntaxhighlight></li>
<li>passing variable over:
Below is just an example - we don’t need to pass <code>my_k</code> since <code>dv</code> will directly get it when we wnat to use it.
<syntaxhighlight lang="python">dv["my_k"] = my_k</syntaxhighlight></li>
<li>we can also use the <code>%px</code> magick to run lines of code on all servers. Given I think <code>numpy</code> is cooler as <code>np</code> I actually have to do it this way, not the <code>dv.sync_imports</code> way.
<syntaxhighlight lang="python">%px import numpy as np</syntaxhighlight></li></ol>

=== actual functionality ===

The actual functionality is very similar to <code>multiprocess</code>/<code>multiprocessing</code> with <code>maps</code> and <code>apply</code>.

==== synchronous ====

<syntaxhighlight lang="python">%%timeit
results = dv.map_sync(my_simulation, my_k)</syntaxhighlight>
<syntaxhighlight lang="python">201 ms ± 34.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)</syntaxhighlight>
==== asynchronous ====

<syntaxhighlight lang="python">%%timeit
results = dv.map_async(my_simulation, my_k).get() # not clear why I needed to use .get() to get something returnable for async</syntaxhighlight>
<syntaxhighlight lang="python">209 ms ± 28.1 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)</syntaxhighlight>
= additional topics and resources =

# [https://ray.io/ <code>Ray</code> python package] allows paralleilization across multiple computers
# [https://people.duke.edu/~ccc14/sta-663-2018/notebooks/S14A_Parallel_Programming_Introduction.html Duke’s statistical computing course] provides a set of examples for multiple python parallelization packages |

Main Page

2025-02-24T15:37:26Z

Skipper:

Anyone can read this wiki.  If you want to write/edit stuff in the wiki, you will need to log in with your account provided by the Statistics & Data Science Department computing group. This Wiki is intended to be a repository for instruction, information and ideas related to the many computing environments available to members and affiliates of the Statistics & Data Science Department.

== Getting to know the computing resources & getting support ==

If you're interested in knowing about what resources you have access to (and/or) what resources you have access to that might solve a problem you're having, please visit our [[What_resources_when|"what resources when" article]].

To get support that is beyond this website please attend office hours below and/or email [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu] for support (for more details about how the support ticketing works through [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu] + ServiceNow please see the [[Service_now|ServiceNow]] article).

Reset your StatDS password here - https://util.stat.cmu.edu/passwordreset/

== Department Contacts ==

• [mailto:events@stat.cmu.edu events@stat.cmu.edu]: seminars, conferences & workshops, and calendar updates

• [mailto:news@stat.cmu.edu news@stat.cmu.edu]: dept/dept member-related public news & announcements

• [mailto:lists@stat.cmu.edu lists@stat.cmu.edu]: department email lists

• [mailto:webmaster@stat.cmu.edu webmaster@stat.cmu.edu]: website updates

• [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu]: computer/tech requests

• [mailto:rooms@stat.cmu.edu rooms@stat.cmu.edu]: request space for TA office hours

• [mailto:reimbursements@stat.cmu.edu reimbursements@stat.cmu.edu]: travel and reimbursement requests

== Computing Office Hours ==

These office hour opportunities provide a dedicated time for you to come ask questions about research computing.  These include code advice, how to access certain resources, what the best resources are for your project, or to learn about new connections to help you progress in your education and research. Staff Office hour times will be sent out each week on Monday as a calendar invite; accept any of the invitations to add it to your personal calendar. PhD TA Office Hours are below, unless a different time is communicated.

#PhD TA Office Hours are held by Alex Shen in Baker 136E on Tuesdays from 3 pm to 5 pm
#Staff Office Hours are held by appointment by Carl Skipper in BH 129G. Schedule via [mailto:lists@stat.cmu.edu email].

See [[Computing_office_hour_usage|computing office hour usage]] for what office hours can be used for.

== Computing Wiki Pages ==

=== General Computing (personal computing) ===

'''Printing'''

*[[Department_Printers|Department Printers]]

'''Virtual environments'''

#[[Virtual Environment Introduction|Why use virtual environments?]]
#R's [[Renv]]
#Python's [[Python_virtual_environments|virtual environments (2 options)]]
#Old instructions for [[Conda with Python (Reference Only)|conda with Python]]

'''[[Parallel_computing|Parallelization]]'''

#[[Parallelization_in_r|R-specific parallelization]]
#[[Parallelization_in_python|Python-specific parallelization]]

'''Package development'''

#[[Developing_R_packages|R package development]]
#[[Developing_python_packages|Python package development]]

'''Version Control'''

See the guide provided in the site of [https://36-750.github.io/tools/version-control/ 36750 - Statistical Computing]

'''Reference management'''

#Zotero
#Mendeley

'''Integrated Development Environments (IDEs)'''

#[[VSCode|VSCode]]
#PyCharm
#RStudio
#Eclipse

'''Containers'''

#[[Introduction_to_containers|Introduction to containers]]
#[[Docker,_and_why-you-should-use-it-for-your-research|Docker, and why-you-should-use-it-for-your-research]]
#[[Tutorials|Basic tutorial]]
#[[Running_Docker_locally_and_on_Hydras|Running Docker locally and on Hydras]]

'''Text Editors'''

#Vim
#Nano
#Emacs
#Atom
#Sublime

See also the guide provided in the site of [https://36-750.github.io/tools/editors/ 36750 - Statistical Computing]

'''Web Hosting'''

#[[How_to_host_your_webpage_under_stat.cmu.edu|How to host your webpage under stat.cmu.edu]]
#[[Bulding_your_website_with_Hugo|Bulding your website with Hugo]]
'''AI Tools'''

* [[ChatGPT and other LLMs]]

*[[github_copilot|GitHub Copilot]]

'''Miscellanea'''

#[[Windows_Subsystem_for_Linux_(WSL)_and_Python|Windows Subsystem for Linux (WSL) and Python]]
#[[Amphetamine|Amphetamine: How to keep your computer awake]]

=== General Department Computing Resources ===

*Fall 2021 [[Computing_seminar_resources|Computing Seminar Resources]] (video and slides)
*[[Linux_and_department_servers|Department Servers]]. This wiki introduces you to the '''hydra''' servers, basic Linux commands, and more.
*#[[Terminal_multiplexers|Terminal multiplexers (screen and tmux)]]. This wiki presents tools for use on the ''hydra'' that can allow you to run jobs even when your not on the server, and much much more!
*#[[Python_on_the_server|Python on the server]]. This wiki introduces uses command line approaches to running <code>python</code> on a virtual machine, and how to use jupyter notebooks through the server as well.
*#[[R_on_the_server|R on the server]]. This wiki introduces uses command line approaches to running <code>R</code> on a virtual machine.
*#[[Terminal_automation|Terminal automation]]. This wiki introduces approaches to automate Linux/OS command line calls.
*##[[Bash_files|Bash files]]. This wiki presents potential uses of Bash files including automated file/folder creation, parallelization and much more!
*##[[Make|Make]]. This wiki presents potential uses of <code>make</code> and Makefiles including automated workflow and dependency tracking
*##[[Jumping_to_frequently_used_folders_and_automating_environment_setups|Jumping to frequently used folders and automating environment setups]]

*[[Department_GPU_Servers|Department GPU Servers]]
*[[GPU_Cluster|GPU Cluster]]
*[[PSC|PSC]]
*#[[ACCESS|ACCESS]]
*#[[Slurm|Slurm]]. This wiki introduces the job management system uses at PSC.
*#[[Parallelization_on_PSC/Bridges-2|Parallelization_on_PSC/Bridges-2]]
*#[[Access_PSC_compute_nodes_directly_from_VSCode|Access PSC compute nodes directly from VSCode]]

*Azure
*#[[Azure_getting_started|Azure: getting started]]. This wiki walks you through the steps to create a virtual machine through Azure
*#[[How_to_obtain_a_virtual_machine_with_a_GPU|How to obtain a virtual machine with a GPU]]
*#[[Attach_a_disk_to_a_virtual_machine|Attach a disk to a virtual machine]]

== Ideas for the Wiki ==

Please see the [[TODO|TODO page]] to post suggestions and errata.

== Using the Wiki ==

*Here is some [[Generic_Wiki_Information|Generic Wiki Information]] that you may be interested in, such as a users' guide and other resources for Wiki users.

Main Page

2025-02-24T15:37:00Z

Skipper:

Anyone can read this wiki.  If you want to write/edit stuff in the wiki, you will need to log in with your account provided by the Statistics & Data Science Department computing group. This Wiki is intended to be a repository for instruction, information and ideas related to the many computing environments available to members and affiliates of the Statistics & Data Science Department.

== Getting to know the computing resources & getting support ==

If you're interested in knowing about what resources you have access to (and/or) what resources you have access to that might solve a problem you're having, please visit our [[What_resources_when|"what resources when" article]].

To get support that is beyond this website please attend office hours below and/or email [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu] for support (for more details about how the support ticketing works through [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu] + ServiceNow please see the [[Service_now|ServiceNow]] article).

Reset your StatDS password here - https://util.stat.cmu.edu/passwordreset/

== Department Contacts ==

• [mailto:events@stat.cmu.edu events@stat.cmu.edu]: seminars, conferences & workshops, and calendar updates

• [mailto:news@stat.cmu.edu news@stat.cmu.edu]: dept/dept member-related public news & announcements

• [mailto:lists@stat.cmu.edu lists@stat.cmu.edu]: department email lists

• [mailto:webmaster@stat.cmu.edu webmaster@stat.cmu.edu]: website updates

• [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu]: computer/tech requests

• [mailto:rooms@stat.cmu.edu rooms@stat.cmu.edu]: request space for TA office hours

• [mailto:reimbursements@stat.cmu.edu reimbursements@stat.cmu.edu]: travel and reimbursement requests

== Computing Office Hours ==

These office hour opportunities provide a dedicated time for you to come ask questions about research computing.  These include code advice, how to access certain resources, what the best resources are for your project, or to learn about new connections to help you progress in your education and research. Staff Office hour times will be sent out each week on Monday as a calendar invite; accept any of the invitations to add it to your personal calendar. PhD TA Office Hours are below, unless a different time is communicated.

#PhD TA Office Hours are held by Alex Shen in Baker 136E on Tuesdays from 3 pm to 5 pm
#Staff Office Hours are held by appointment by Carl Skipper in BH 129G. Schedule via .

See [[Computing_office_hour_usage|computing office hour usage]] for what office hours can be used for.

== Computing Wiki Pages ==

=== General Computing (personal computing) ===

'''Printing'''

*[[Department_Printers|Department Printers]]

'''Virtual environments'''

#[[Virtual Environment Introduction|Why use virtual environments?]]
#R's [[Renv]]
#Python's [[Python_virtual_environments|virtual environments (2 options)]]
#Old instructions for [[Conda with Python (Reference Only)|conda with Python]]

'''[[Parallel_computing|Parallelization]]'''

#[[Parallelization_in_r|R-specific parallelization]]
#[[Parallelization_in_python|Python-specific parallelization]]

'''Package development'''

#[[Developing_R_packages|R package development]]
#[[Developing_python_packages|Python package development]]

'''Version Control'''

See the guide provided in the site of [https://36-750.github.io/tools/version-control/ 36750 - Statistical Computing]

'''Reference management'''

#Zotero
#Mendeley

'''Integrated Development Environments (IDEs)'''

#[[VSCode|VSCode]]
#PyCharm
#RStudio
#Eclipse

'''Containers'''

#[[Introduction_to_containers|Introduction to containers]]
#[[Docker,_and_why-you-should-use-it-for-your-research|Docker, and why-you-should-use-it-for-your-research]]
#[[Tutorials|Basic tutorial]]
#[[Running_Docker_locally_and_on_Hydras|Running Docker locally and on Hydras]]

'''Text Editors'''

#Vim
#Nano
#Emacs
#Atom
#Sublime

See also the guide provided in the site of [https://36-750.github.io/tools/editors/ 36750 - Statistical Computing]

'''Web Hosting'''

#[[How_to_host_your_webpage_under_stat.cmu.edu|How to host your webpage under stat.cmu.edu]]
#[[Bulding_your_website_with_Hugo|Bulding your website with Hugo]]
'''AI Tools'''

* [[ChatGPT and other LLMs]]

*[[github_copilot|GitHub Copilot]]

'''Miscellanea'''

#[[Windows_Subsystem_for_Linux_(WSL)_and_Python|Windows Subsystem for Linux (WSL) and Python]]
#[[Amphetamine|Amphetamine: How to keep your computer awake]]

=== General Department Computing Resources ===

*Fall 2021 [[Computing_seminar_resources|Computing Seminar Resources]] (video and slides)
*[[Linux_and_department_servers|Department Servers]]. This wiki introduces you to the '''hydra''' servers, basic Linux commands, and more.
*#[[Terminal_multiplexers|Terminal multiplexers (screen and tmux)]]. This wiki presents tools for use on the ''hydra'' that can allow you to run jobs even when your not on the server, and much much more!
*#[[Python_on_the_server|Python on the server]]. This wiki introduces uses command line approaches to running <code>python</code> on a virtual machine, and how to use jupyter notebooks through the server as well.
*#[[R_on_the_server|R on the server]]. This wiki introduces uses command line approaches to running <code>R</code> on a virtual machine.
*#[[Terminal_automation|Terminal automation]]. This wiki introduces approaches to automate Linux/OS command line calls.
*##[[Bash_files|Bash files]]. This wiki presents potential uses of Bash files including automated file/folder creation, parallelization and much more!
*##[[Make|Make]]. This wiki presents potential uses of <code>make</code> and Makefiles including automated workflow and dependency tracking
*##[[Jumping_to_frequently_used_folders_and_automating_environment_setups|Jumping to frequently used folders and automating environment setups]]

*[[Department_GPU_Servers|Department GPU Servers]]
*[[GPU_Cluster|GPU Cluster]]
*[[PSC|PSC]]
*#[[ACCESS|ACCESS]]
*#[[Slurm|Slurm]]. This wiki introduces the job management system uses at PSC.
*#[[Parallelization_on_PSC/Bridges-2|Parallelization_on_PSC/Bridges-2]]
*#[[Access_PSC_compute_nodes_directly_from_VSCode|Access PSC compute nodes directly from VSCode]]

*Azure
*#[[Azure_getting_started|Azure: getting started]]. This wiki walks you through the steps to create a virtual machine through Azure
*#[[How_to_obtain_a_virtual_machine_with_a_GPU|How to obtain a virtual machine with a GPU]]
*#[[Attach_a_disk_to_a_virtual_machine|Attach a disk to a virtual machine]]

== Ideas for the Wiki ==

Please see the [[TODO|TODO page]] to post suggestions and errata.

== Using the Wiki ==

*Here is some [[Generic_Wiki_Information|Generic Wiki Information]] that you may be interested in, such as a users' guide and other resources for Wiki users.

ACCESS

2025-01-30T19:42:19Z

Skipper:

The NSF’s ACCESS (Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support) program builds upon the successes of the 11-year XSEDE project.

NSF funded supercomputers, like PSC are available via the ACCESS federated system.  Besides parallel or memory-intensive computing options, there are many GPU resources available to ACCESS users via the Credit Marketplace.

#If you are a Graduate Student, all you need is '''an account''',''' '''your''' CV''', a '''1-paragraph abstract''' and '''a letter from your advisor'''.  With these 4 items, you can complete a simple form and get up to 400,000 cpu-hours to exchange for access to most of the nations supercomputing cyberinfrastructure. Checkout the following [https://allocations.access-ci.org/preparing-your-explore-access-request link].  
#If you previously held an XSEDE account, you can login here: [https://identity.access-ci.org/new-user-federated https://identity.access-ci.org/new-user-federated] and complete your account transition from XSEDE to ACCESS by selecting “ACCESS CI (XSEDE)” and subsequently login using Andrew Web Login.  You will then be guided through a short verification process similar to any other website.  
#If you are '''new''' to the department or supercomputing in general, I set up a new account so you will be able to access (no pun intended) these resources easily now or in the future.  Visit [https://identity.access-ci.org/new-user-federated https://identity.access-ci.org/new-user-federated] to set up a new account -  make sure to choose “'''Carnegie Mellon University'''”!  
#The interface to the Supercomputing Credit Market is still being improved, but we are working on documentation to address that.

This will shortly be updated to add easy access to ACCESS On-Ramps: <div id="resource-catalog-react"></div>

How to obtain a virtual machine with a GPU

2024-11-14T19:54:25Z

Skipper:

This page is ARCHIVED. Our agreement with Microsoft to gain Azure virtual machines is over and you should explore our other GPU options for the department. These include our [[Department GPU Servers]] and the shared Dietrich [[GPU Cluster]].

<nowiki>-------------------------------------------------------------------------------------------------------------------------------------------------------------------------</nowiki>

The following is the process to request access to a virtual machine (VM) with GPU on Azure

#Determine the Azure Service and VM Family size you are looking for and calculate the total CPU quantity needed. (For example, a Standard_NC6s_v3 has 1 GPU (Nvidia V100) but 6 CPU, so you would request 6 CPU)fff
#From within the subscription, start the CPU Quota increase request process and ensure the appropriate Azure service is specified. (Ex: Virtual Machine, Azure Machine Learning, Azure Lab Services)
#Work with Azure Support on the request.
#If Support denies the request, e-mail Carl or the PhD Computing TA and they will give you the email of the right point of contact. He will escalate your Support Incident number to our internal contact and work with Azure Support on getting the request approved.
#Requests that get approved are typically fulfilled within 48 hours of escalation.

Steps 1 and 2 are detailed in the figure below. Step 3 entails communicating via email with a Microsoft Support Engineer.

'''Please note: The NCasT4v3, ND A100 v4, NDM A100 v4, and NDv2 are constrained globally, and we are not able to support sponsorship requests for these SKUs for the foreseeable future.''' Please also keep in mind that due to constraints, the number of GPUs and the region you select may need to be adjusted depending on the demand in the region. In some instances, choosing a different region, VM Family, and/or different quantity of GPUs may yield a different result.

In the (usually rare) case that the deployment of the approved VM does not go through completely, note that there is a known issue that prevents the CUDA extension to be installed correctly. If this happens, you can install it manually by following this [https://learn.microsoft.com/en-us/azure/virtual-machines/linux/n-series-driver-setup guide]. Since this is a quite involved process, if you find yourself stuck we suggest to go to the PhD computing office hours and/or contact Carl.

[[File:3200px-Azure vm quota request.png|1600x1000px|Azure vm quota request.png]]

How to obtain a virtual machine with a GPU

2024-11-14T19:50:42Z

Skipper:

This page is ARCHIVED. Our agreement with Microsoft to gain Azure virtual machines is over and you should explore our other GPU options for the department.

The following is the process to request access to a virtual machine (VM) with GPU on Azure

#Determine the Azure Service and VM Family size you are looking for and calculate the total CPU quantity needed. (For example, a Standard_NC6s_v3 has 1 GPU (Nvidia V100) but 6 CPU, so you would request 6 CPU)fff
#From within the subscription, start the CPU Quota increase request process and ensure the appropriate Azure service is specified. (Ex: Virtual Machine, Azure Machine Learning, Azure Lab Services)
#Work with Azure Support on the request.
#If Support denies the request, e-mail Carl or the PhD Computing TA and they will give you the email of the right point of contact. He will escalate your Support Incident number to our internal contact and work with Azure Support on getting the request approved.
#Requests that get approved are typically fulfilled within 48 hours of escalation.

Steps 1 and 2 are detailed in the figure below. Step 3 entails communicating via email with a Microsoft Support Engineer.

'''Please note: The NCasT4v3, ND A100 v4, NDM A100 v4, and NDv2 are constrained globally, and we are not able to support sponsorship requests for these SKUs for the foreseeable future.''' Please also keep in mind that due to constraints, the number of GPUs and the region you select may need to be adjusted depending on the demand in the region. In some instances, choosing a different region, VM Family, and/or different quantity of GPUs may yield a different result.

In the (usually rare) case that the deployment of the approved VM does not go through completely, note that there is a known issue that prevents the CUDA extension to be installed correctly. If this happens, you can install it manually by following this [https://learn.microsoft.com/en-us/azure/virtual-machines/linux/n-series-driver-setup guide]. Since this is a quite involved process, if you find yourself stuck we suggest to go to the PhD computing office hours and/or contact Carl.

[[File:3200px-Azure vm quota request.png|1600x1000px|Azure vm quota request.png]]

How to obtain a virtual machine with a GPU

2024-05-24T02:41:50Z

Skipper:

The following is the process to request access to a virtual machine (VM) with GPU on Azure

#Determine the Azure Service and VM Family size you are looking for and calculate the total CPU quantity needed. (For example, a Standard_NC6s_v3 has 1 GPU (Nvidia V100) but 6 CPU, so you would request 6 CPU)fff
#From within the subscription, start the CPU Quota increase request process and ensure the appropriate Azure service is specified. (Ex: Virtual Machine, Azure Machine Learning, Azure Lab Services)
#Work with Azure Support on the request.
#If Support denies the request, e-mail Carl or the PhD Computing TA and they will give you the email of the right point of contact. He will escalate your Support Incident number to our internal contact and work with Azure Support on getting the request approved.
#Requests that get approved are typically fulfilled within 48 hours of escalation.

Steps 1 and 2 are detailed in the figure below. Step 3 entails communicating via email with a Microsoft Support Engineer.

'''Please note: The NCasT4v3, ND A100 v4, NDM A100 v4, and NDv2 are constrained globally, and we are not able to support sponsorship requests for these SKUs for the foreseeable future.''' Please also keep in mind that due to constraints, the number of GPUs and the region you select may need to be adjusted depending on the demand in the region. In some instances, choosing a different region, VM Family, and/or different quantity of GPUs may yield a different result.

In the (usually rare) case that the deployment of the approved VM does not go through completely, note that there is a known issue that prevents the CUDA extension to be installed correctly. If this happens, you can install it manually by following this [https://learn.microsoft.com/en-us/azure/virtual-machines/linux/n-series-driver-setup guide]. Since this is a quite involved process, if you find yourself stuck we suggest to go to the PhD computing office hours and/or contact Carl.

[[File:3200px-Azure vm quota request.png|1600x1000px|Azure vm quota request.png]]

File:3200px-Azure vm quota request.png

2024-05-24T02:41:32Z

Skipper:

test

Azure getting started

2024-05-24T02:39:32Z

Skipper:

= Azure =

== Respectiful use of Azure resources ==

Ensure you’re not wasting precious department Azure credits - when you are not using your virtual machine, please shut it down (and then relaunch it when you need to use it). You can shut-down and relauch a server on the resource page. The overview for your virtual machine might look something like the image below. To turn off your machine click the “Stop” button on the “Overview” tab, and to launch your machine click the “Start” button. Every time you launch your machine it’s likely that the IP address will have changed, so use the “Connect” button to figure out the new IP address (though it is also listed on the Overview page). The “Connect” page (which appears in the second image) will also remind you of what your username for the virtual machine is.

[[File:1600px-Azure on off.png|frameless|800px|turning on and off server to save department money]]

[[File:1600px-Azure connect.png|frameless|800px|connecting to your server's IP address when it changes]]

= Getting started: create an Azure virtual machine =

== Getting an account on portal.azure.com ==

The first step is to complete [https://docs.google.com/forms/d/1ZrynhmPAyLAS-TlIN5iPcQSPBbwzVJe1MPbqgYYejRs this form]  asking for an account on Azure. This should prompt a set of emails from Carl. In this process, you’ll be (1) creating an account on portal.azure.com with your @andrew.cmu.edu email address and (2) getting connected to the statistics department resources.

=== Debugging account access ===

If you log onto portal.azure.com and it suggests that you are not connected to the statistics department (it will offer free trials), then you are not correctly connected. This may be due to errors in either of the above.

To check if you’re able to connect to the department resources, go to the top right corner (with your andrew email address) and click it. You’ll see an option “Switch directory”. Click it. The hope is in the potential startup directories, you’ll see one called “Default Directory” as seen in the image below.

[[File:Azure user on portal.png|frameless|200px|User account on portal.azure.com]]

 

[[File:1600px-Azure directory options.png|frameless|800px|understanding and debugging directory options (starting out)]]

== Creating a Resource ==

Creating a virtual on Azure can be a bit complicated (which is why other resources like the department servers are available for certain types of computing).

The first thing to do is click the “create a resource” button on your azure home page (should look something like this:)

[[File:Azure create new.png|frameless|300px|create new server button]]

You can also do so in the <code>All resources</code> > <code>+ Create</code>

 

----

For statistical computing we select a resource from the <code>AI + Machine Learning</code> tab on the left side and then select <code>Data Science Virtual Machine - Ubuntu 18.04</code> for a linux virtual machine. This selection will look something like the image below:

[[File:Azure select resource type.png|frameless|700px|Selecting which type of virtual server to use]]

=== Defining your Resource ===

There are a lot of options to create your resource, but you can ignore a lot of these.

==== Basics ====

Below is 2 images of the first “Basics” page. You will need to

#fill out / select your “Resource group”
#define your “Virtual machine name” (might we recommend (username)-(CPU/GPU)-cores(#)-(RAM)-(index). For example “bpleroy-GPU-cores4-27-1”.
#select the size of the machine (relative to it’s cores and types of cores). You will see available options. If you do not see an option that meets your needs, please message remarks@stat.cmu.edu / go see Carl and describe your question. You can also click "see all images" to explore a list of options (some which are not available to CMU Stats). If you are looking for GPU in this list filter by "family" and "GPU" for the family.
#finally, we recommend using a ssh public key (which will be downloaded at the end of the set-up process). We recommend you change your “Username” to your andrew id but you can leave it as “azureuser” if you’d like (we recommend such things since you may forget and it's natural to have it consistent across your servers).

[[File:1400px-Azure basics 1.png|frameless|700px|setting up new server, basic page (1)]]

[[File:1400px-Azure basics 2.png|frameless|700px|setting up new server, basic page (2)]]

==== Disks ====

The second pages is about “Disks” although creating storage on Azure is complicated it’s a very necessary procedure. Click “Create and attache a new disk”. You will then have to select the right size. Generally, disks smaller than 512 GB are cheap for the department but 1 TB + are more costly. You personal computer likely has between 128 - 512 GB (maybe 1TB), but it’s really unlikely you’ll need too much for your project.

[[File:1400px-Azure disk creation.png|frameless|700px|creating a disk]]

 

==== Network and Management ====

There isn’t really anything necessary for you to do on these tabs.

==== Advanced ====

If you are using GPUs you’ll need to add the NVDIA extension for GPUs.

==== Tags are also not necessary ====

==== Done? ====

You did it! Save your ssh key (<code>.pem</code> file) in a location where you plan to call the server from. Also note that you will '''still''' need to correctly link your data disk you just created (which is discussed in the next step).

=== Connecting your data disk ===

This process is slightly complex (and can have implications if you don’t do it correctly). We recommend you follow the [https://docs.microsoft.com/en-us/azure/virtual-machines/linux/attach-disk-portal azure docs].

==== step 1: finding and identifying your disk ====

on the server run something like:

 
<syntaxhighlight lang="bash">lsblk -o NAME,HCTL,SIZE,MOUNTPOINT | grep -i "sd"</syntaxhighlight>

you should see a list something like this:

 
<syntaxhighlight lang="bash">sda 3:0:0:0 128G
sdb 0:0:0:0 150G
├─sdb1 149.9G /
├─sdb14 4M
└─sdb15 106M /boot/efi
sdc 1:0:1:0 176G
└─sdc1 176G /mnt</syntaxhighlight>

now, on portal for the same VM, look at your disks (left side: Settings > Disks) and examine the size of your disk you added (for mine it’s 128G). As such we can conclude that my disk that needs mounting is <code>sda</code> (this might not be the same for you - and is not the same in the azure docs).

==== step 2: partitioning a new disk ====

Copied form the azure docs (but changed for my disk name)

 
<syntaxhighlight lang="bash">sudo parted /dev/sda --script mklabel gpt mkpart xfspart xfs 0% 100%
sudo mkfs.xfs /dev/sda1
sudo partprobe /dev/sda1</syntaxhighlight>

==== step 3: mount the disk ====

Create a directory to mount the file system using mkdir. The following example creates a directory at <code>/datadrive</code>:

 
<syntaxhighlight lang="bash">sudo mkdir /datadrive</syntaxhighlight>

Use mount to then mount the filesystem. The following example mounts the <code>/dev/sda1</code> partition to the <code>/datadrive</code> mount point:

 
<syntaxhighlight lang="bash">sudo mount /dev/sda1 /datadrive</syntaxhighlight>

''Now we have'':

 
<syntaxhighlight lang="bash">lsblk -o NAME,HCTL,SIZE,MOUNTPOINT | grep -i "sd"</syntaxhighlight>
<syntaxhighlight lang="bash">sda 3:0:0:0 128G
└─sda1 128G /datadrive
sdb 0:0:0:0 150G
├─sdb1 149.9G /
├─sdb14 4M
└─sdb15 106M /boot/efi
sdc 1:0:1:0 176G
└─sdc1 176G /mnt</syntaxhighlight>

==== step 4: … ====

 
<syntaxhighlight lang="bash">sudo blkid</syntaxhighlight>
<syntaxhighlight lang="bash">/dev/sdb1: LABEL="cloudimg-rootfs" UUID="1fe704c3-9cec-4cb5-b634-2e46f6bd9f90" TYPE="ext4" PARTUUID="318ec118-9248-4203-9cd5-7cb8c36f04d6"
/dev/sdb15: LABEL="UEFI" UUID="52A4-029B" TYPE="vfat" PARTUUID="2332d2b4-222e-425d-866e-227b42f3979d"
/dev/sdc1: UUID="9cc09cc7-c65d-4fe4-a397-75679a438c1c" TYPE="ext4" PARTUUID="2b584de2-01"
/dev/sda1: UUID="a2ba271d-e5a4-4416-84d2-b9408864b53b" TYPE="xfs" PARTLABEL="xfspart" PARTUUID="77452866-0b43-4485-9956-673377498608"</syntaxhighlight>

==== step 5: creating alias link to datadrive ====

To create a symbolic link to your a <code>workspace</code> folder to the <code>/datadrive</code> you can do the following.

 
<syntaxhighlight lang="bash">ln -s /datadrive ~/workspace </syntaxhighlight>

Afterwards, everything in the <code>workspace</code> will actually be stored on your new <code>/datadrive</code>.

==== step 6: making sure you have correct permissions ====

<code>ls -al /</code> (might show you don’t have access (write, read, execute) to the /datadrive), so we did

<code>sudo chmod 777 datadrive</code> …

[https://www.hostinger.com/tutorials/how-to-create-symbolic-links-in-linux/ https://www.hostinger.com/tutorials/how-to-create-symbolic-links-in-linux/]

=== Connecting to your server and moving files ===

To connect to your server you will need to know (1) where you're ssh key is (often called <code>_server_name_.pem</code> (this should have been downloaded at the end of the creation of your server). '''Note that the first time your connect you need to make sure you have the correct permissions - see the connect page for the correct <code>cmod</code> command you need to use!''' (2) your username defined in the [[Azure_getting_started#Basics|Basics]] step

You will also need to know what IP address your server is currently using. This can before on the connect page of your server (as seen in the second image in this article). Then you're type something like

 
<syntaxhighlight lang="bash">
ssh -i _my_ssh_key_.pem _user_name_@_IP_address_
</syntaxhighlight>

To pass files to and from your server with <code>scp</code>, we will imagine you are on your home computer. Then do as follows:

#'''server to local''' <syntaxhighlight lang="bash">
scp -i _my_ssh_key_.pem _user_name_@_IP_address_:/full/path/to/files/on/server/my_script.R /local/path/to/files/on/local/computer/
</syntaxhighlight>

#'''local to server''' <syntaxhighlight lang="bash">
scp -i _my_ssh_key_.pem /local/path/to/files/on/local/computer/my_script.R _user_name_@_IP_address_:/full/path/to/files/
</syntaxhighlight>

To pass directories (and do the full thing you'll need to do the same as above but add

 
<syntaxhighlight lang="bash">
scp -i _my_ssh_key.pem -r ...
</syntaxhighlight>

Azure getting started

2024-05-24T02:37:27Z

Skipper:

= Azure =

== Respectiful use of Azure resources ==

Ensure you’re not wasting precious department Azure credits - when you are not using your virtual machine, please shut it down (and then relaunch it when you need to use it). You can shut-down and relauch a server on the resource page. The overview for your virtual machine might look something like the image below. To turn off your machine click the “Stop” button on the “Overview” tab, and to launch your machine click the “Start” button. Every time you launch your machine it’s likely that the IP address will have changed, so use the “Connect” button to figure out the new IP address (though it is also listed on the Overview page). The “Connect” page (which appears in the second image) will also remind you of what your username for the virtual machine is.

[[File:1600px-Azure on off.png|frameless|800px|turning on and off server to save department money]]

[[File:1600px-Azure connect.png|frameless|800px|connecting to your server's IP address when it changes]]

= Getting started: create an Azure virtual machine =

== Getting an account on portal.azure.com ==

The first step is to complete [https://docs.google.com/forms/d/1ZrynhmPAyLAS-TlIN5iPcQSPBbwzVJe1MPbqgYYejRs this form]  asking for an account on Azure. This should prompt a set of emails from Carl. In this process, you’ll be (1) creating an account on portal.azure.com with your @andrew.cmu.edu email address and (2) getting connected to the statistics department resources.

=== Debugging account access ===

If you log onto portal.azure.com and it suggests that you are not connected to the statistics department (it will offer free trials), then you are not correctly connected. This may be due to errors in either of the above.

To check if you’re able to connect to the department resources, go to the top right corner (with your andrew email address) and click it. You’ll see an option “Switch directory”. Click it. The hope is in the potential startup directories, you’ll see one called “Default Directory” as seen in the image below.

[[File:Azure user on portal.png|frameless|200px|User account on portal.azure.com]]

 

[[File:Azure directory options.png|frameless|800px|understanding and debugging directory options (starting out)]]

== Creating a Resource ==

Creating a virtual on Azure can be a bit complicated (which is why other resources like the department servers are available for certain types of computing).

The first thing to do is click the “create a resource” button on your azure home page (should look something like this:)

[[File:Azure create new.png|frameless|300px|create new server button]]

You can also do so in the <code>All resources</code> > <code>+ Create</code>

 

----

For statistical computing we select a resource from the <code>AI + Machine Learning</code> tab on the left side and then select <code>Data Science Virtual Machine - Ubuntu 18.04</code> for a linux virtual machine. This selection will look something like the image below:

[[File:Azure select resource type.png|frameless|700px|Selecting which type of virtual server to use]]

=== Defining your Resource ===

There are a lot of options to create your resource, but you can ignore a lot of these.

==== Basics ====

Below is 2 images of the first “Basics” page. You will need to

#fill out / select your “Resource group”
#define your “Virtual machine name” (might we recommend (username)-(CPU/GPU)-cores(#)-(RAM)-(index). For example “bpleroy-GPU-cores4-27-1”.
#select the size of the machine (relative to it’s cores and types of cores). You will see available options. If you do not see an option that meets your needs, please message remarks@stat.cmu.edu / go see Carl and describe your question. You can also click "see all images" to explore a list of options (some which are not available to CMU Stats). If you are looking for GPU in this list filter by "family" and "GPU" for the family.
#finally, we recommend using a ssh public key (which will be downloaded at the end of the set-up process). We recommend you change your “Username” to your andrew id but you can leave it as “azureuser” if you’d like (we recommend such things since you may forget and it's natural to have it consistent across your servers).

[[File:1400px-Azure basics 1.png|frameless|700px|setting up new server, basic page (1)]]

[[File:1400px-Azure basics 2.png|frameless|700px|setting up new server, basic page (2)]]

==== Disks ====

The second pages is about “Disks” although creating storage on Azure is complicated it’s a very necessary procedure. Click “Create and attache a new disk”. You will then have to select the right size. Generally, disks smaller than 512 GB are cheap for the department but 1 TB + are more costly. You personal computer likely has between 128 - 512 GB (maybe 1TB), but it’s really unlikely you’ll need too much for your project.

[[File:1400px-Azure disk creation.png|frameless|700px|creating a disk]]

 

==== Network and Management ====

There isn’t really anything necessary for you to do on these tabs.

==== Advanced ====

If you are using GPUs you’ll need to add the NVDIA extension for GPUs.

==== Tags are also not necessary ====

==== Done? ====

You did it! Save your ssh key (<code>.pem</code> file) in a location where you plan to call the server from. Also note that you will '''still''' need to correctly link your data disk you just created (which is discussed in the next step).

=== Connecting your data disk ===

This process is slightly complex (and can have implications if you don’t do it correctly). We recommend you follow the [https://docs.microsoft.com/en-us/azure/virtual-machines/linux/attach-disk-portal azure docs].

==== step 1: finding and identifying your disk ====

on the server run something like:

 
<syntaxhighlight lang="bash">lsblk -o NAME,HCTL,SIZE,MOUNTPOINT | grep -i "sd"</syntaxhighlight>

you should see a list something like this:

 
<syntaxhighlight lang="bash">sda 3:0:0:0 128G
sdb 0:0:0:0 150G
├─sdb1 149.9G /
├─sdb14 4M
└─sdb15 106M /boot/efi
sdc 1:0:1:0 176G
└─sdc1 176G /mnt</syntaxhighlight>

now, on portal for the same VM, look at your disks (left side: Settings > Disks) and examine the size of your disk you added (for mine it’s 128G). As such we can conclude that my disk that needs mounting is <code>sda</code> (this might not be the same for you - and is not the same in the azure docs).

==== step 2: partitioning a new disk ====

Copied form the azure docs (but changed for my disk name)

 
<syntaxhighlight lang="bash">sudo parted /dev/sda --script mklabel gpt mkpart xfspart xfs 0% 100%
sudo mkfs.xfs /dev/sda1
sudo partprobe /dev/sda1</syntaxhighlight>

==== step 3: mount the disk ====

Create a directory to mount the file system using mkdir. The following example creates a directory at <code>/datadrive</code>:

 
<syntaxhighlight lang="bash">sudo mkdir /datadrive</syntaxhighlight>

Use mount to then mount the filesystem. The following example mounts the <code>/dev/sda1</code> partition to the <code>/datadrive</code> mount point:

 
<syntaxhighlight lang="bash">sudo mount /dev/sda1 /datadrive</syntaxhighlight>

''Now we have'':

 
<syntaxhighlight lang="bash">lsblk -o NAME,HCTL,SIZE,MOUNTPOINT | grep -i "sd"</syntaxhighlight>
<syntaxhighlight lang="bash">sda 3:0:0:0 128G
└─sda1 128G /datadrive
sdb 0:0:0:0 150G
├─sdb1 149.9G /
├─sdb14 4M
└─sdb15 106M /boot/efi
sdc 1:0:1:0 176G
└─sdc1 176G /mnt</syntaxhighlight>

==== step 4: … ====

 
<syntaxhighlight lang="bash">sudo blkid</syntaxhighlight>
<syntaxhighlight lang="bash">/dev/sdb1: LABEL="cloudimg-rootfs" UUID="1fe704c3-9cec-4cb5-b634-2e46f6bd9f90" TYPE="ext4" PARTUUID="318ec118-9248-4203-9cd5-7cb8c36f04d6"
/dev/sdb15: LABEL="UEFI" UUID="52A4-029B" TYPE="vfat" PARTUUID="2332d2b4-222e-425d-866e-227b42f3979d"
/dev/sdc1: UUID="9cc09cc7-c65d-4fe4-a397-75679a438c1c" TYPE="ext4" PARTUUID="2b584de2-01"
/dev/sda1: UUID="a2ba271d-e5a4-4416-84d2-b9408864b53b" TYPE="xfs" PARTLABEL="xfspart" PARTUUID="77452866-0b43-4485-9956-673377498608"</syntaxhighlight>

==== step 5: creating alias link to datadrive ====

To create a symbolic link to your a <code>workspace</code> folder to the <code>/datadrive</code> you can do the following.

 
<syntaxhighlight lang="bash">ln -s /datadrive ~/workspace </syntaxhighlight>

Afterwards, everything in the <code>workspace</code> will actually be stored on your new <code>/datadrive</code>.

==== step 6: making sure you have correct permissions ====

<code>ls -al /</code> (might show you don’t have access (write, read, execute) to the /datadrive), so we did

<code>sudo chmod 777 datadrive</code> …

[https://www.hostinger.com/tutorials/how-to-create-symbolic-links-in-linux/ https://www.hostinger.com/tutorials/how-to-create-symbolic-links-in-linux/]

=== Connecting to your server and moving files ===

To connect to your server you will need to know (1) where you're ssh key is (often called <code>_server_name_.pem</code> (this should have been downloaded at the end of the creation of your server). '''Note that the first time your connect you need to make sure you have the correct permissions - see the connect page for the correct <code>cmod</code> command you need to use!''' (2) your username defined in the [[Azure_getting_started#Basics|Basics]] step

You will also need to know what IP address your server is currently using. This can before on the connect page of your server (as seen in the second image in this article). Then you're type something like

 
<syntaxhighlight lang="bash">
ssh -i _my_ssh_key_.pem _user_name_@_IP_address_
</syntaxhighlight>

To pass files to and from your server with <code>scp</code>, we will imagine you are on your home computer. Then do as follows:

#'''server to local''' <syntaxhighlight lang="bash">
scp -i _my_ssh_key_.pem _user_name_@_IP_address_:/full/path/to/files/on/server/my_script.R /local/path/to/files/on/local/computer/
</syntaxhighlight>

#'''local to server''' <syntaxhighlight lang="bash">
scp -i _my_ssh_key_.pem /local/path/to/files/on/local/computer/my_script.R _user_name_@_IP_address_:/full/path/to/files/
</syntaxhighlight>

To pass directories (and do the full thing you'll need to do the same as above but add

 
<syntaxhighlight lang="bash">
scp -i _my_ssh_key.pem -r ...
</syntaxhighlight>

File:1400px-Azure disk creation.png

2024-05-24T02:36:56Z

Skipper:

test

File:1400px-Azure basics 2.png

2024-05-24T02:36:21Z

Skipper:

test

File:1400px-Azure basics 1.png

2024-05-24T02:35:37Z

Skipper:

test

File:Azure select resource type.png

2024-05-24T02:34:52Z

Skipper:

test

File:Azure create new.png

2024-05-24T02:34:12Z

Skipper:

test

File:1600px-Azure directory options.png

2024-05-24T02:33:27Z

Skipper:

test

File:Azure user on portal.png

2024-05-24T02:32:32Z

Skipper:

Azure user portal

Azure getting started

2024-05-24T02:31:13Z

Skipper:

= Azure =

== Respectiful use of Azure resources ==

Ensure you’re not wasting precious department Azure credits - when you are not using your virtual machine, please shut it down (and then relaunch it when you need to use it). You can shut-down and relauch a server on the resource page. The overview for your virtual machine might look something like the image below. To turn off your machine click the “Stop” button on the “Overview” tab, and to launch your machine click the “Start” button. Every time you launch your machine it’s likely that the IP address will have changed, so use the “Connect” button to figure out the new IP address (though it is also listed on the Overview page). The “Connect” page (which appears in the second image) will also remind you of what your username for the virtual machine is.

[[File:1600px-Azure on off.png|frameless|800px|turning on and off server to save department money]]

[[File:1600px-Azure connect.png|frameless|800px|connecting to your server's IP address when it changes]]

= Getting started: create an Azure virtual machine =

== Getting an account on portal.azure.com ==

The first step is to complete [https://docs.google.com/forms/d/1ZrynhmPAyLAS-TlIN5iPcQSPBbwzVJe1MPbqgYYejRs this form]  asking for an account on Azure. This should prompt a set of emails from Carl. In this process, you’ll be (1) creating an account on portal.azure.com with your @andrew.cmu.edu email address and (2) getting connected to the statistics department resources.

=== Debugging account access ===

If you log onto portal.azure.com and it suggests that you are not connected to the statistics department (it will offer free trials), then you are not correctly connected. This may be due to errors in either of the above.

To check if you’re able to connect to the department resources, go to the top right corner (with your andrew email address) and click it. You’ll see an option “Switch directory”. Click it. The hope is in the potential startup directories, you’ll see one called “Default Directory” as seen in the image below.

[[File:Azure user on portal.png|frameless|200px|User account on portal.azure.com]]

 

[[File:Azure directory options.png|frameless|800px|understanding and debugging directory options (starting out)]]

== Creating a Resource ==

Creating a virtual on Azure can be a bit complicated (which is why other resources like the department servers are available for certain types of computing).

The first thing to do is click the “create a resource” button on your azure home page (should look something like this:)

[[File:Azure create new.png|frameless|300px|create new server button]]

You can also do so in the <code>All resources</code> > <code>+ Create</code>

 

----

For statistical computing we select a resource from the <code>AI + Machine Learning</code> tab on the left side and then select <code>Data Science Virtual Machine - Ubuntu 18.04</code> for a linux virtual machine. This selection will look something like the image below:

[[File:Azure select resource type.png|frameless|700px|Selecting which type of virtual server to use]]

=== Defining your Resource ===

There are a lot of options to create your resource, but you can ignore a lot of these.

==== Basics ====

Below is 2 images of the first “Basics” page. You will need to

#fill out / select your “Resource group”
#define your “Virtual machine name” (might we recommend (username)-(CPU/GPU)-cores(#)-(RAM)-(index). For example “bpleroy-GPU-cores4-27-1”.
#select the size of the machine (relative to it’s cores and types of cores). You will see available options. If you do not see an option that meets your needs, please message remarks@stat.cmu.edu / go see Carl and describe your question. You can also click "see all images" to explore a list of options (some which are not available to CMU Stats). If you are looking for GPU in this list filter by "family" and "GPU" for the family.
#finally, we recommend using a ssh public key (which will be downloaded at the end of the set-up process). We recommend you change your “Username” to your andrew id but you can leave it as “azureuser” if you’d like (we recommend such things since you may forget and it's natural to have it consistent across your servers).

[[File:Azure basics 1.png|frameless|700px|setting up new server, basic page (1)]]

[[File:Azure basics 2.png|frameless|700px|setting up new server, basic page (2)]]

==== Disks ====

The second pages is about “Disks” although creating storage on Azure is complicated it’s a very necessary procedure. Click “Create and attache a new disk”. You will then have to select the right size. Generally, disks smaller than 512 GB are cheap for the department but 1 TB + are more costly. You personal computer likely has between 128 - 512 GB (maybe 1TB), but it’s really unlikely you’ll need too much for your project.

[[File:Azure disk creation.png|frameless|700px|creating a disk]]

 

==== Network and Management ====

There isn’t really anything necessary for you to do on these tabs.

==== Advanced ====

If you are using GPUs you’ll need to add the NVDIA extension for GPUs.

==== Tags are also not necessary ====

==== Done? ====

You did it! Save your ssh key (<code>.pem</code> file) in a location where you plan to call the server from. Also note that you will '''still''' need to correctly link your data disk you just created (which is discussed in the next step).

=== Connecting your data disk ===

This process is slightly complex (and can have implications if you don’t do it correctly). We recommend you follow the [https://docs.microsoft.com/en-us/azure/virtual-machines/linux/attach-disk-portal azure docs].

==== step 1: finding and identifying your disk ====

on the server run something like:

 
<syntaxhighlight lang="bash">lsblk -o NAME,HCTL,SIZE,MOUNTPOINT | grep -i "sd"</syntaxhighlight>

you should see a list something like this:

 
<syntaxhighlight lang="bash">sda 3:0:0:0 128G
sdb 0:0:0:0 150G
├─sdb1 149.9G /
├─sdb14 4M
└─sdb15 106M /boot/efi
sdc 1:0:1:0 176G
└─sdc1 176G /mnt</syntaxhighlight>

now, on portal for the same VM, look at your disks (left side: Settings > Disks) and examine the size of your disk you added (for mine it’s 128G). As such we can conclude that my disk that needs mounting is <code>sda</code> (this might not be the same for you - and is not the same in the azure docs).

==== step 2: partitioning a new disk ====

Copied form the azure docs (but changed for my disk name)

 
<syntaxhighlight lang="bash">sudo parted /dev/sda --script mklabel gpt mkpart xfspart xfs 0% 100%
sudo mkfs.xfs /dev/sda1
sudo partprobe /dev/sda1</syntaxhighlight>

==== step 3: mount the disk ====

Create a directory to mount the file system using mkdir. The following example creates a directory at <code>/datadrive</code>:

 
<syntaxhighlight lang="bash">sudo mkdir /datadrive</syntaxhighlight>

Use mount to then mount the filesystem. The following example mounts the <code>/dev/sda1</code> partition to the <code>/datadrive</code> mount point:

 
<syntaxhighlight lang="bash">sudo mount /dev/sda1 /datadrive</syntaxhighlight>

''Now we have'':

 
<syntaxhighlight lang="bash">lsblk -o NAME,HCTL,SIZE,MOUNTPOINT | grep -i "sd"</syntaxhighlight>
<syntaxhighlight lang="bash">sda 3:0:0:0 128G
└─sda1 128G /datadrive
sdb 0:0:0:0 150G
├─sdb1 149.9G /
├─sdb14 4M
└─sdb15 106M /boot/efi
sdc 1:0:1:0 176G
└─sdc1 176G /mnt</syntaxhighlight>

==== step 4: … ====

 
<syntaxhighlight lang="bash">sudo blkid</syntaxhighlight>
<syntaxhighlight lang="bash">/dev/sdb1: LABEL="cloudimg-rootfs" UUID="1fe704c3-9cec-4cb5-b634-2e46f6bd9f90" TYPE="ext4" PARTUUID="318ec118-9248-4203-9cd5-7cb8c36f04d6"
/dev/sdb15: LABEL="UEFI" UUID="52A4-029B" TYPE="vfat" PARTUUID="2332d2b4-222e-425d-866e-227b42f3979d"
/dev/sdc1: UUID="9cc09cc7-c65d-4fe4-a397-75679a438c1c" TYPE="ext4" PARTUUID="2b584de2-01"
/dev/sda1: UUID="a2ba271d-e5a4-4416-84d2-b9408864b53b" TYPE="xfs" PARTLABEL="xfspart" PARTUUID="77452866-0b43-4485-9956-673377498608"</syntaxhighlight>

==== step 5: creating alias link to datadrive ====

To create a symbolic link to your a <code>workspace</code> folder to the <code>/datadrive</code> you can do the following.

 
<syntaxhighlight lang="bash">ln -s /datadrive ~/workspace </syntaxhighlight>

Afterwards, everything in the <code>workspace</code> will actually be stored on your new <code>/datadrive</code>.

==== step 6: making sure you have correct permissions ====

<code>ls -al /</code> (might show you don’t have access (write, read, execute) to the /datadrive), so we did

<code>sudo chmod 777 datadrive</code> …

[https://www.hostinger.com/tutorials/how-to-create-symbolic-links-in-linux/ https://www.hostinger.com/tutorials/how-to-create-symbolic-links-in-linux/]

=== Connecting to your server and moving files ===

To connect to your server you will need to know (1) where you're ssh key is (often called <code>_server_name_.pem</code> (this should have been downloaded at the end of the creation of your server). '''Note that the first time your connect you need to make sure you have the correct permissions - see the connect page for the correct <code>cmod</code> command you need to use!''' (2) your username defined in the [[Azure_getting_started#Basics|Basics]] step

You will also need to know what IP address your server is currently using. This can before on the connect page of your server (as seen in the second image in this article). Then you're type something like

 
<syntaxhighlight lang="bash">
ssh -i _my_ssh_key_.pem _user_name_@_IP_address_
</syntaxhighlight>

To pass files to and from your server with <code>scp</code>, we will imagine you are on your home computer. Then do as follows:

#'''server to local''' <syntaxhighlight lang="bash">
scp -i _my_ssh_key_.pem _user_name_@_IP_address_:/full/path/to/files/on/server/my_script.R /local/path/to/files/on/local/computer/
</syntaxhighlight>

#'''local to server''' <syntaxhighlight lang="bash">
scp -i _my_ssh_key_.pem /local/path/to/files/on/local/computer/my_script.R _user_name_@_IP_address_:/full/path/to/files/
</syntaxhighlight>

To pass directories (and do the full thing you'll need to do the same as above but add

 
<syntaxhighlight lang="bash">
scp -i _my_ssh_key.pem -r ...
</syntaxhighlight>

File:1600px-Azure on off.png

2024-05-24T02:30:55Z

Skipper:

test

File:1600px-Azure connect.png

2024-05-24T02:30:15Z

Skipper:

test

Amphetamine

2024-05-24T02:22:33Z

Skipper:

Modern laptops implement several functionalities to reduce power usage, e.g. by putting drives to sleep, dimming the display, or going into sleep or hibernate modes. This is especially true for Macs.

'''What if you need your laptop to stay awake for a fixed period of time for particular reasons?'''

For example, what if you have a job running locally either from the terminal or from a Python notebook in VSCode? '''Enter Amphetamine!'''

== What is Amphetamine? ==
[[File:1000px-Amphetamine.png|500x500px|Amphetamine.png]]

Amphetamine is a '''free''' utility by William Gustafson that lets you click, schedule, or trigger Mac alertness from the menu bar. 

In the simplest case, you can choose an item from the menu: Indefinitely keeps your Mac awake until canceled, but you can also select a duration, while a given app remains running, or until a file is downloaded. The last one, for example, works by asking you to pick an observation interval after which Amphetamine checks if the file has changed in size so it can predict that the file has completed its download.

For '''research purposes''', Amphetamine comes particularly handy when we need a process to keep running in the background without having to monitor the laptop constantly to avoid it going to sleep. This happens especially often with '''VSCode''' when connected to a remote machine via '''SSH'''. If a '''Jupyter notebook''' is running through a remote ssh connection, any cell running will be interrupted if the connection drops, which happens if the Mac goes to sleep. Amphetamine easily prevents this issue: it's enought to start a session indefintely and stop it when the job has finished. 

Keep an eye on your battery consumption though! The price to pay for Amphetamine sessions (especially those that prevent any kind of sleep mode) is increased battery consumption. 

== Amphetamine Enhancer ==

Amphetamine Enhancer is a helper application that enhances a couple of Amphetamine's features. In its 1.0 release, Amphetamine Enhancer adds the following abilities to Amphetamine:

• Adds a fail-safe for the '''closed-display mode''' feature found in Amphetamine 5.0+ • Gives Amphetamine 5.0+ the ability to use all running processes with Triggers and app-based sessions

== Download ==

For Mac users, the app is freely available on the [https://apps.apple.com/us/app/amphetamine/id937984704?mt=12 App Store]. Windows users can download Amphetamine from the [https://apps.microsoft.com/detail/9NBLGGH4WHSP?hl=en-US&gl=US Microsoft Store].

Amphetamine Enhancer is freely available for download on [https://github.com/x74353/Amphetamine-Enhancer Github].

Check this [https://zoomadmin.com/HowToInstall/UbuntuPackage/amphetamine article] for a guide on how to install Amphetamine for Ubuntu.

File:1000px-Amphetamine.png

2024-05-24T02:22:14Z

Skipper:

test

Tutorials

2024-05-24T02:19:16Z

Skipper:

This tutorial is supposed to serve as a simple self-contained example that shows how to build a Docker image for a project that might be of interest in our department: a simple application using PyTorch.

To complete this tutorial, we will rely on [[Python_virtual_environments|conda]] Python version 3.8 or later. You also need to have installed Docker (see [https://docs.docker.com/desktop/ download and install Docker]).

== Example application ==

Create a directory named <code>docker </code>and create a conda environment with Python and PyTorch installed in it. Don't forget to activate the environment before installing packages!

Then, create a <code>test_docker.py </code>file and add the following simple lines:
<pre>import torch

a = torch.Tensor([1, 2, 3, 4, 5])

if __name__ == '__main__':
print(a)
</pre>

= Inherit from an existing image =

== Create a Dockerfile for Python ==

Inside the <code>docker </code>directory create a <code>Dockerfile</code> (simply by tipying <code>touch Dockerfile</code>) and add a line that tells Docker what base image to use for the application.
<pre># syntax=docker/dockerfile:1

FROM python:3.8-slim-buster</pre>

Docker images can inherit from other images. Therefore, instead of creating your own base image, you can use the official Python image that has all the tools and packages needed to run a Python application.

To make things easier when running the remaining commands, create a working directory. This instructs Docker to use this path as the default location for all subsequent commands. This means you can use relative file paths based on the working directory instead of full file paths.
<pre>WORKDIR /path/to/your/project/dir
</pre>

Now we would like the Docker image to install all the necessary packages we have in the environment. When installing Python, conda automatically downloads the <code>pip </code>package manager as well. We can then use it to create a requirements file by typing
<pre>pip list --format=freeze > requirements.txt</pre>

in the terminal, within the <code>docker</code> directory. Then we need to make sure the requirements file will be in the image. The <code>COPY</code> command takes two parameters. The first parameter tells Docker what file(s) you would like to copy into the image. The second parameter tells Docker where to copy that file(s) to. For this example, copy the <code>requirements.txt</code> file into the working directory:
<pre>COPY requirements.txt requirements.txt
</pre>

With the <code>requirements.txt</code> file inside the image, you can use the <code>RUN</code> command to run <code>pip install</code>. This works exactly the same as running <code>pip install</code> locally on a machine, but this time pip installs the modules into the image.
<pre>RUN pip install -r requirements.txt
</pre>

At this point, you have an image based on Python version 3.8 and have installed the dependencies. The next step is to add the source code into the image. Use the <code>COPY</code> command as with the <code>requirements.txt</code> file. 
<pre>COPY test_docker.py test_docker.py</pre>

Now, tell Docker what command to run when the image is executed inside a container using the <code>CMD</code> command.
<pre>CMD ["python", "test_docker.py"]</pre>

The complete<code> Dockerfile </code>should look like this:
<pre># syntax=docker/dockerfile:1
FROM python:3.8-slim-buster
WORKDIR /path/to/your/project/dir
COPY requirements.txt requirements.txt
RUN pip install -r requirements.txt
COPY test_docker.py test_docker.py
CMD ["python", "test_docker.py"]
</pre>

== Build an image ==

Now that you’ve created the Dockerfile, let’s build the image. To do this, use the <code>docker build</code> command. The <code>docker build</code> command builds Docker images from a Dockerfile and a “context”. A build’s context is the set of files located in the specified PATH or URL. The Docker build process can access any of the files located in this context.

The build command optionally takes a <code>--tag</code> flag. The tag sets the name of the image and an optional tag in the format <code>name:tag. </code>Don't forget to open the Docker Desktop application before running the build command, otherwise the Docker deamon will not be running and the command will not work.

If everything went well, you should see something like this:

'''<code>[[File:1704px-Docker build.png|900x500px|RTENOTITLE]]</code>'''

== Run the image ==

Now that we have an image, we can run that image and see if our application is running correctly.

A container is a normal operating system process except that this process is isolated in that it has its own file system, its own networking, and its own isolated process tree separate from the host.

To run an image inside of a container, we use the <code>docker run</code> command. The <code>docker run</code> command requires one parameter which is the name of the image. Let’s start our image and make sure it is running correctly. Run the following command in your terminal.
<pre>docker run test-docker</pre>

If everything goes well, you should see the desired output printed in the terminal (in this case, a simple PyTorch Tensor).

= Creating new base images =

Most Dockerfiles start from a parent image. If you need to completely control the contents of your image, you might need to create a base image instead. Here’s the difference:

*A [https://docs.docker.com/glossary/#parent-image parent image] is the image that your image is based on. It refers to the contents of the <code>FROM</code> directive in the Dockerfile. Each subsequent declaration in the Dockerfile modifies this parent image. Most Dockerfiles start from a parent image, rather than a base image. However, the terms are sometimes used interchangeably.
*A [https://docs.docker.com/glossary/#base-image base image] has <code>FROM scratch</code> in its Dockerfile.

See this [https://docs.docker.com/build/building/base-images/ link] for details.

File:1704px-Docker build.png

2024-05-24T02:18:59Z

Skipper:

test

VSCode

2024-05-24T02:12:22Z

Skipper:

VSCode is an open-source IDE maintained by Microsoft, and it is considered a “lightweight” code editor. In comparison with full integrated development environment (IDE) editors which occupy gigabytes of disk space, VS Code uses less than 200MB when installed. Despite the “lightweight” term, Visual Studio Code offers a massive number of features (via so-called “extensions”) which keep increasing and improving with every new update. 

Resources:

#[https://code.visualstudio.com/docs Official documentation]
#[https://code.visualstudio.com/download Download]

 

== Basic Usage ==

Some good extensions: 

#Python (and Pylance)
#Jupyter
#R
#LaTeX Workshop
#Remote - SSH
#autoDocstring

== Remote Development and Prototyping via SSH ==

In this section we will learn how to use VSCode to manage, develop, and test code that we have stored on a remote server (like [https://www.stat.cmu.edu/computewiki/index.php/Linux_and_department_servers Hydras], [https://www.stat.cmu.edu/computewiki/index.php/Azure_getting_started Azure], etc...).

This is particularly useful to avoid painful workflows that see us doing the following:

#develop locally on our laptop
#scp some scripts to the remote server and run them (assuming we have consistent environments)
#scp the results to the local laptop
#repeat from 1

Ideally, we want to be able to use the remote server only as the computational workhorse, while leveraging the GUI of our local operative system. Doing this in VSCode is extremely simple:

#In the left toolbar, click on the “Extensions” button and search for SSH. The correct one is hte one developed by Microsoft (blue icon). Click on the install button.

[[File:Ssh.png|1000x1000px|RTENOTITLE]]

 
<ol start="2">
<li>Once the extension is installed, click on the small green button at the very bottom-left of the window (the one with a symbol similar to ><). A pop-up window at the top will open: click on “Connect to Host”</li>
<li>At this point, we have two alternatives:
# add a new SSH host manually, which will prompt another small window where to enter the connection details (similar to what one does from the terminal)
#or '''better''', if we have a SSH <code>config </code>file in the <code>.ssh </code>directory, VSCode will automatically list the possible remote servers that are specified in this file. If we specified a path to a key, then VSCode will connect automatically. If instead the remote server requires a password (as it is the case for Hydras), then we will be prompted with another window to enter it. As an example:
[[File:Hosts.png|RTENOTITLE]]
 [[File:Ssh config.png|RTENOTITLE]]
<ol start="4">
<li>Congratulations! You are now connected to a remote server through VSCode. By clicking on the “Explorer” button at the top in the left-side toolbar, you can navigate the file system of the server, develop and run code, etc. Everything you do and see is actually being hosted, run, and saved remotely. 
[[File:Remote file.png|1000x1000px|RTENOTITLE]]</li>
</ol>
</li>
</ol>

 

== Managing environments within VSCode ==

Since each project might (and should) have its own development environment that you setup, with dependencies and packages that make it unique for the tasks of this project, VSCode allows to easily choose which interpreter to use. 

We simply have to click at the bottom right of the window, on the button that shows the current Python (or other language) version that we are using. This will open a prompt to choose the environment among those present on the machine we are developing on:

[[File:Env.png|1000x1000px|Env.png]]

 

== Support for Jupyter Notebooks (even remote!) within VSCode ==

In one of the previous section we learned how to setup VSCode for remote development. A great feature of the workflow that we have just enabled is that we can run Jupyter Notebooks (and produce plots, tables, etc...) that are hosted remotely while seeing the output on our local laptop within VSCode (this works also if we are running things locally). Doing so is very easy:

#Click on the “Extensions” button on the left toolbar and download the Jupyter extension

[[File:Jupyter.png|1000x1000px|Jupyter.png]]
<ol start="2">
<li>Open a .ipynb file and choose the kernel (interpreter for the correct project environment) from the button that appears at the top right of the opened notebook</li>
</ol>

[[File:Choose kernel.png|1000x1000px|Choose kernel.png]]
<ol start="3">
<li>Happy coding!</li>
</ol>

== Support for RStudio (even remote!) within VSCode ==

Install the R extension, which provides support for the [https://www.r-project.org/ R programming language], including features such as extended syntax highlighting, R language service based on code analysis, interacting with R terminals, viewing data, plots, workspace variables, help pages, managing packages, and working with [https://rmarkdown.rstudio.com/ R Markdown] documents.

== Miscellanea ==

=== Amphetamine ===

If you need your laptop to stay awake while there's a process running --- you are running some cell in a notebook via remote SSH --- check out this [[Amphetamine|article in the wiki]].

=== Tunnelling ===

Disclaimer: this is particularly useful for PSC (Pittsburgh Supercomuting Center) users, as it allows to connect directly to a compute node on PSC and have things like Jupyter Notebooks run via VSCode without problems. This solution bypasses a problem incurred when connecting via SSH from VSCode, which caused to land on the login nodes. These have very little memory and basically do not support any computation.

Instructions for creating a VS code tunnel to PSC:

*First set up code CLI on PSC: [https://dev.to/burkeholland/introducing-vs-code-remote-tunnels-connect-to-remote-machines-with-ease-3nlg https://dev.to/burkeholland/introducing-vs-code-remote-tunnels-connect-to-remote-machines-with-ease-3nlg ]
*Then run the following:
<pre> curl -Lk 'https://code.visualstudio.com/sha/download?build=stable&os=cli-alpine-x64' --output vscode_cli.tar.gz
tar -xf vscode_cli.tar.gz
</pre>

           This will create a "code" executable file which we need to add to the path. Open <code>.bashrc</code> (or <code>.zshrc </code>if you have a recent Mac) and edit the line that exports the path to be 
<pre>export PATH="/jet/home/<username>:$PATH"</pre>

*Restart the shell and check that "which code" finds the installation.
*Next, install the remote tunnels and remote ssh extensions on VS code.
**These are found in the extensions menu
**After installation there should be a blue "open remote window" (>< symbol) in the bottom left of the VS code window.

*Next, we set up the tunnel: [https://dev.to/burkeholland/introducing-vs-code-remote-tunnels-connect-to-remote-machines-with-ease-3nlg https://dev.to/burkeholland/introducing-vs-code-remote-tunnels-connect-to-remote-machines-with-ease-3nlg ]
**Start interactive job
**Run <code>code tunnel</code> and follow the instructions. I recommend logging in via GitHub the first time and occasionally you will be asked to name the machine. By default it will name it based off of the node you are currently interacting with; however, the tunnel will connect to any node at PSC via the same setup so I recommend you name it something generic related to PSC. You will eventually see a message like “Open this link in your browser https://vscode.dev/tunnel/<machine name>”
**On VS Code on your local machine, click the blue "open remote window" (>< symbol) in the bottom left and select Connect to Tunnel.
**Select GitHub and then the machine that matches <machine name> in the link shown on the remote location. This should reload your VS code window to be running on the remote server, but if you started an interactive job it is running on a compute node.
**To stop the connection, run "ctrl + c"
**If you need to deregister the tunnel run "code tunnel unregister" however, this should be unnecessary

For any future connection you run this same set of steps, as I haven't found a way to avoid re logging in via github every time.

File:Choose kernel.png

2024-05-24T02:12:11Z

Skipper:

test

File:Jupyter.png

2024-05-24T02:11:47Z

Skipper:

test

File:Env.png

2024-05-24T02:11:24Z

Skipper:

Env

File:Remote file.png

2024-05-24T02:10:34Z

Skipper:

RTENOTITLE

File:Hosts.png

2024-05-24T02:09:34Z

Skipper:

RTENOTITLE

VSCode

2024-05-24T02:08:31Z

Skipper:

VSCode is an open-source IDE maintained by Microsoft, and it is considered a “lightweight” code editor. In comparison with full integrated development environment (IDE) editors which occupy gigabytes of disk space, VS Code uses less than 200MB when installed. Despite the “lightweight” term, Visual Studio Code offers a massive number of features (via so-called “extensions”) which keep increasing and improving with every new update. 

Resources:

#[https://code.visualstudio.com/docs Official documentation]
#[https://code.visualstudio.com/download Download]

 

== Basic Usage ==

Some good extensions: 

#Python (and Pylance)
#Jupyter
#R
#LaTeX Workshop
#Remote - SSH
#autoDocstring

== Remote Development and Prototyping via SSH ==

In this section we will learn how to use VSCode to manage, develop, and test code that we have stored on a remote server (like [https://www.stat.cmu.edu/computewiki/index.php/Linux_and_department_servers Hydras], [https://www.stat.cmu.edu/computewiki/index.php/Azure_getting_started Azure], etc...).

This is particularly useful to avoid painful workflows that see us doing the following:

#develop locally on our laptop
#scp some scripts to the remote server and run them (assuming we have consistent environments)
#scp the results to the local laptop
#repeat from 1

Ideally, we want to be able to use the remote server only as the computational workhorse, while leveraging the GUI of our local operative system. Doing this in VSCode is extremely simple:

#In the left toolbar, click on the “Extensions” button and search for SSH. The correct one is hte one developed by Microsoft (blue icon). Click on the install button.

[[File:Ssh.png|1000x1000px|RTENOTITLE]]

 
<ol start="2">
<li>Once the extension is installed, click on the small green button at the very bottom-left of the window (the one with a symbol similar to ><). A pop-up window at the top will open: click on “Connect to Host”</li>
<li>At this point, we have two alternatives:
# add a new SSH host manually, which will prompt another small window where to enter the connection details (similar to what one does from the terminal)
#or '''better''', if we have a SSH <code>config </code>file in the <code>.ssh </code>directory, VSCode will automatically list the possible remote servers that are specified in this file. If we specified a path to a key, then VSCode will connect automatically. If instead the remote server requires a password (as it is the case for Hydras), then we will be prompted with another window to enter it. As an example:[[File:Ssh config.png|RTENOTITLE]]
<ol start="4">
<li>Congratulations! You are now connected to a remote server through VSCode. By clicking on the “Explorer” button at the top in the left-side toolbar, you can navigate the file system of the server, develop and run code, etc. Everything you do and see is actually being hosted, run, and saved remotely. [[File:Remote file.png|1000x1000px|Remote file.png]]</li>
</ol>
</li>
</ol>

 

== Managing environments within VSCode ==

Since each project might (and should) have its own development environment that you setup, with dependencies and packages that make it unique for the tasks of this project, VSCode allows to easily choose which interpreter to use. 

We simply have to click at the bottom right of the window, on the button that shows the current Python (or other language) version that we are using. This will open a prompt to choose the environment among those present on the machine we are developing on:

[[File:Env.png|1000x1000px|Env.png]]

 

== Support for Jupyter Notebooks (even remote!) within VSCode ==

In one of the previous section we learned how to setup VSCode for remote development. A great feature of the workflow that we have just enabled is that we can run Jupyter Notebooks (and produce plots, tables, etc...) that are hosted remotely while seeing the output on our local laptop within VSCode (this works also if we are running things locally). Doing so is very easy:

#Click on the “Extensions” button on the left toolbar and download the Jupyter extension

[[File:Jupyter.png|1000x1000px|Jupyter.png]]
<ol start="2">
<li>Open a .ipynb file and choose the kernel (interpreter for the correct project environment) from the button that appears at the top right of the opened notebook</li>
</ol>

[[File:Choose kernel.png|1000x1000px|Choose kernel.png]]
<ol start="3">
<li>Happy coding!</li>
</ol>

== Support for RStudio (even remote!) within VSCode ==

Install the R extension, which provides support for the [https://www.r-project.org/ R programming language], including features such as extended syntax highlighting, R language service based on code analysis, interacting with R terminals, viewing data, plots, workspace variables, help pages, managing packages, and working with [https://rmarkdown.rstudio.com/ R Markdown] documents.

== Miscellanea ==

=== Amphetamine ===

If you need your laptop to stay awake while there's a process running --- you are running some cell in a notebook via remote SSH --- check out this [[Amphetamine|article in the wiki]].

=== Tunnelling ===

Disclaimer: this is particularly useful for PSC (Pittsburgh Supercomuting Center) users, as it allows to connect directly to a compute node on PSC and have things like Jupyter Notebooks run via VSCode without problems. This solution bypasses a problem incurred when connecting via SSH from VSCode, which caused to land on the login nodes. These have very little memory and basically do not support any computation.

Instructions for creating a VS code tunnel to PSC:

*First set up code CLI on PSC: [https://dev.to/burkeholland/introducing-vs-code-remote-tunnels-connect-to-remote-machines-with-ease-3nlg https://dev.to/burkeholland/introducing-vs-code-remote-tunnels-connect-to-remote-machines-with-ease-3nlg ]
*Then run the following:
<pre> curl -Lk 'https://code.visualstudio.com/sha/download?build=stable&os=cli-alpine-x64' --output vscode_cli.tar.gz
tar -xf vscode_cli.tar.gz
</pre>

           This will create a "code" executable file which we need to add to the path. Open <code>.bashrc</code> (or <code>.zshrc </code>if you have a recent Mac) and edit the line that exports the path to be 
<pre>export PATH="/jet/home/<username>:$PATH"</pre>

*Restart the shell and check that "which code" finds the installation.
*Next, install the remote tunnels and remote ssh extensions on VS code.
**These are found in the extensions menu
**After installation there should be a blue "open remote window" (>< symbol) in the bottom left of the VS code window.

*Next, we set up the tunnel: [https://dev.to/burkeholland/introducing-vs-code-remote-tunnels-connect-to-remote-machines-with-ease-3nlg https://dev.to/burkeholland/introducing-vs-code-remote-tunnels-connect-to-remote-machines-with-ease-3nlg ]
**Start interactive job
**Run <code>code tunnel</code> and follow the instructions. I recommend logging in via GitHub the first time and occasionally you will be asked to name the machine. By default it will name it based off of the node you are currently interacting with; however, the tunnel will connect to any node at PSC via the same setup so I recommend you name it something generic related to PSC. You will eventually see a message like “Open this link in your browser https://vscode.dev/tunnel/<machine name>”
**On VS Code on your local machine, click the blue "open remote window" (>< symbol) in the bottom left and select Connect to Tunnel.
**Select GitHub and then the machine that matches <machine name> in the link shown on the remote location. This should reload your VS code window to be running on the remote server, but if you started an interactive job it is running on a compute node.
**To stop the connection, run "ctrl + c"
**If you need to deregister the tunnel run "code tunnel unregister" however, this should be unnecessary

For any future connection you run this same set of steps, as I haven't found a way to avoid re logging in via github every time.

File:Ssh config.png

2024-05-24T02:08:16Z

Skipper:

RTENOTITLE

File:Ssh.png

2024-05-24T02:04:53Z

Skipper:

RTENOTITLE

Developing python packages

2024-05-24T02:00:16Z

Skipper:

This wiki presents a motivational example of a lot of properties of python packages and how to use them. Please see [[packages motivation]] for motivation about why you should utilize the package framework while doing research. This material captures the basics presented by [https://py-pkgs.org/ py-pkgs], (recently) developed by UBC professors. As such you’ll find that python packages online may have different structures. Still, <code>py-pkgs</code> provides a lot of helpful tools to automate python package development. The <code>py-pkgs</code> project was actually started to mirror the helpful [[Developing_R_packages|package development tools in <code>R</code>]]. Note that this wiki page does provide additional commentary but the process proposed is very similar to <code>py-pkgs</code>’s approach.

== Foundations (what tools will be used) ==

In the package development pipeline that is proposed by <code>py-pkgs</code> we have a few very power packages to help you develop an <code>python</code> package as painlessly as possible.

* <code>poetry</code> help build a python package
* <code>cookiecutter</code> helps utilize pre-made templates (for our package’s files)
* <code>pytest</code> provides clean and standardized tools for testing
* (numpy style code) provides rubrics for documentation

== Building an python package ==

This process is slightly different than the one we demonstrated in the [[Developing_R_packages]] wiki (we set up the structure of the package and then link to github).

=== Step 1: Creating a minimal source package ===

A python (''source'') package just files in a directory, formatted in a specific way.

To create the bare bones python package, we will be using the packages <code>poetry</code> and <code>cookiecutter</code>, so if you haven’t already done so, install those two packages (we’ll be using <code>conda</code>) by typing:

<syntaxhighlight lang="bash">conda install -c conda-forge poetry cookiecutter</syntaxhighlight>
''Note that this line is taken directly from [https://py-pkgs.org/02-setup py-pkgs/02-setup], and it is possible that some day these packages can be installed wit ha simple <code>conda install poetry cookiecutter</code>.''

'''Now''', to actually start setting up your python package we will first use <code>cookiecutter</code> to copy a package structure provided by <code>py-pkgs</code> by running the following in our terminal:

<syntaxhighlight lang="bash">cookiecutter https://github.com/UBC-MDS/cookiecutter-ubc-mds.git</syntaxhighlight>
This will prompt you to answer a set of questions including
<ol>
<li> the author name (can include caps and spaces) </li>
<li> github username </li>
<li> package name (can include caps and spaces) </li>
<li> package slug name (can included underscores but no spaces) </li>
<li> a short package description (can include caps and spaces) </li>
<li> version number (we recommend either 0.0.1 or 0.1.0) </li>
<li> python version 8. open source license (they list the options) </li>
<li> if the package should include github actions (we suggest no for now, and plan to have wikis help with this later) </li>
</ol>

''Note: our package for this example will be named <code>tartanpy</code>'' (an a copy of the "final" product can be found on [https://github.com/benjaminleroy/tartanpy|github].

''Note: the cookiecutter template from <code>py-pkg</code>s (aka the one above) is still under construction - so it might look slightly different in the future.''

This will create a <code>tartanpy</code> folder with the following items:

# Directory <code>src/</code> (where all the function source code goes)
# Directory <code>docs/</code> (where all the pretty looking documentation goes '''- auto generated''')
# Directory <code>tests/</code> (where you’ll store your test, with a placeholder test file)
# <code>pyproject.toml</code> file (a description file about your package, which includes the things you entered when you ran the <code>cookiecutter</code> call)
# <code>LICENSE</code> (booler plate License - don’t really need to deal with this)
# A set of <code>.md</code> files for document things like general info for viewers (<code>README.md</code>), info for contributors and their conduct (<code>CONTRIBUTING.md</code> and <code>CONDUCT.md</code>) and a file to include information on changes you’ve made <code>CHANGELOG.md</code>).

[[File:1200px-Tartanpy folder.png|300px|frame|left|initial directory items]]

==== pyproject.toml file meta-data ====

There are few ways for packages to store their meta-data, and <code>py-pkgs</code>’s path is through the <code>pyproject.toml</code>. Other options include things like <code>setup.py</code> files. If you made a mistake when answering questions via your cookiecutter prompts you can change them by opening up the <code>pyproject.toml</code> file. Our example package’s <code>pyproject.toml</code> file looks like the image below

[[File:1200px-Tartanpy pyproject description.png|300px|frame|left|initialized description file]]

In a <code>pyproject.toml</code> file you’ll notice that each section is headed by a <code>[info]</code> heading. Notice that three of the sections in the image above have <code>tool.poetry</code> headers. This means that the package <code>poetry</code> can update these sections. As we showcase the power of this package you may find that using poetry to update a lot of these fields is more desirable than manually doing it yourself.

=== Step 2: Setting up a Github repo ===

Version control (and discussed in Stat Computing) is a very useful tool to help streamline file management and provide piece of mind. Oftentimes, you will want to have a copy of the repository not just on your personal computer but online (to help with sharing, just in case your computer dies, it’s cool, etc.). As such these steps will walk you through how to set up a github repo for your new package and how to connect that repo to a new python package. This step will mirror thoughts in [https://py-pkgs.org/03-how-to-package-a-python#set-up-remote-version-control py-pkgs/03-how-to-package-a-python].

<ol style="list-style-type: decimal;">
<li>'''In <code>Github</code>''', do the following
<ul>
<li>go to the top-level directory (i.e., <code>github.com/<your user name></code>)</li>
<li>click on “+” at top right, and select “New repository”</li>
<li>name the repository (e.g., “tartan_python”). Note that if you’d like to share your package with others via pypi you’ll need to make your package name unique!</li>
<li>provide a short description of the repository (don’t leave completely blank!)</li>
<li>keep the repository public (as students you have access to free private repos https://education.github.com/pack, but for purposes of this example we will keep the repo public)</li>
<li>'''do not''' initalize with a README, .gitignore, or license (It even says “Skip this step if you’re importing an exisiting repo”!!!)</li>
<li>click on “Create Repository”</li></ul>
</li>
<li>'''…then in your terminal, do the following'''
<ul>
<li>First, initialize your repository with

<syntaxhighlight lang="bash">git init</syntaxhighlight>
then add all <code>cookiecutter</code> files and create your initial commit with <code>bash git add . git commit -m "initial package setup"</code>
</li>
<li>setup connection to github by adding a a remote, your main branch and pushing your first commit with

<syntaxhighlight lang="bash">git remote add origin https://github.com/benjaminleroy/tartanpy.git
git branch -M main
git push -u origin main</syntaxhighlight>
''Note that github provides you with walk throughs that mirror what was recommended above and showcases 2 approaches if you want to use the command line (<code>https</code> and <code>ssh</code>) as well as how you might do it with github desktop. So helpful - thanks github!''</li></ul>

</li></ol>

=== Step 3: Writing a function ===

All of your python code needs to go in <code>.py</code> files in the <code>src/tartanpy/</code> directory. Suppose our first set of functions deal with communicating to the user - then we might make a <code>communicate.py</code> file in <code>src/tartanpy/</code> folder. Inside this file, let’s image we add the following function

<syntaxhighlight lang="python">def call_scotty_demo(your_name):

name_length = len(your_name)

# say Hiiiiii
hi_string = "\n" +\
"-" * (name_length + 8) + "\n" +\
"| Hi " + your_name + "! | \n" +\
"| " + "-" * (name_length + 4) + "\n" +\
"| /\n"

scotty_string = ""+\
" |/ |\\_/| \n" +\
" | |q p| /} \n" +\
" | ( 0 )\"\"\"\\ \n" +\
" \\ |\"-\"` | \n" +\
" || /=\\\\ | \n" +\
" \"'\" '\"\"\"'"

print(hi_string + scotty_string)
</syntaxhighlight>
To get access to this function we can use <code>poetry</code> to locally install our package with the bash command below (that needs to be run in the top folder of our package)

<pre>poetry install</pre>
''Note that without poetry this could be done with a <code>pip install -e .</code> in the same directory and this would actually not require running this line of code every time we update the package. On the other hand - we’ll show the versatility of <code>poetry</code> in the rest of the article.''

To utilize our function we can then open up a interactive python instance (e.g. <code>ipython</code> in your terminal) and then run

<syntaxhighlight lang="python">In [1]: from tartanpy import communicate

In [2]: communicate.call_scotty_demo("Andrew Carnegie")</syntaxhighlight>
which will return:

<syntaxhighlight lang="bash">-----------------------
| Hi Andrew Carnegie! |
| -------------------
| /
|/ |\_/|
| |q p| /}
| ( 0 )"""\
\ |"-"` |
|| /=\\ |
"'" '"""'</syntaxhighlight>
Note that <code>poetry install</code> can be called over and over if you keep updating your function files and will re-load the newest verion of all your functions.

==== Additional notes: ====

===== Understanding <code>__init__.py</code> =====

For <code>python</code> packages you’ll always have an <code>__init__.py</code> file in this directory. We currently have to import the <code>communicate</code> file from <code>tartanpy</code> in order to get access to our amazing function. To make this function come directly with the package, in the <code>__init__.py</code> file we can write

<syntaxhighlight lang="python">from .communicate import call_scotty_demo</syntaxhighlight>
''Note the dot in <code>.communicate</code> says “in this same directory as this current file”, and we don’t have to include the <code>.py</code> file link at the end.'' Additionally This approach '''only''' imports call_scotty_demo to the global package, we could use <code>... import *</code> to import all functions (but this approach is smart if we had helper functions we didn’t really need to share with the end user).*

With the above change, we can then do (the <code>poetry</code> part still needs to be at the top directory):

<syntaxhighlight lang="bash">poetry install
ipython</syntaxhighlight>
and then in the ipython shell:

<syntaxhighlight lang="python">In [1]: import tartanpy

In [2]: tartanpy.call_scotty_demo("Andrew Mellon")</syntaxhighlight>
which will return:

<syntaxhighlight lang="bash">---------------------
| Hi Andrew Mellon! |
| -----------------
| /
|/ |\_/|
| |q p| /}
| ( 0 )"""\
\ |"-"` |
|| /=\\ |
"'" '"""'</syntaxhighlight>
===== Using external packages =====

[You'll want to read the next section as well for a full understanding of dealing with external packages.]

Suppose I wished to truncate really long names I’m greeting but also give the user a warning that I’m going to do so. I’d need to use the <code>warnings</code> package. To use this in our package I’d need to change our <code>communicate.py</code> file to

<syntaxhighlight lang="python">import warnings

def call_scotty_demo(your_name):
if len(your_name) > 30:
warnings.warn("your name is beyond 30 characters, "+\
"has been truncated")
your_name = your_name[:30]

name_length = len(your_name)

# say Hiiiiii
hi_string = "\n" +\
"-" * (name_length + 8) + "\n" +\
"| Hi " + your_name + "! | \n" +\
"| " + "-" * (name_length + 4) + "\n" +\
"| /\n"

scotty_string = ""+\
" |/ |\\_/| \n" +\
" | |q p| /} \n" +\
" | ( 0 )\"\"\"\\ \n" +\
" \\ |\"-\"` | \n" +\
" || /=\\\\ | \n" +\
" \"'\" '\"\"\"'"

print(hi_string + scotty_string)</syntaxhighlight>
Notice that we have to impot the <code>warnings</code> package at the top of our function file.

Now we have the additional funcitonality (after reloading with <code>poetry</code>)

<syntaxhighlight lang="python">In [1]: import tartanpy

In [2]: tartanpy.call_scotty_demo("Oski Bear, your a wonderful, but slightly creepy Berkeley Bear")</syntaxhighlight>
which returns

<syntaxhighlight lang="python">.../tartanpy/src/tartanpy/communicate.py:6: UserWarning: your name is beyond 30 characters, has been truncated
warnings.warn("your name is beyond 30 characters, "+\

--------------------------------------
| Hi Oski Bear, your a wonderful, b! |
| ----------------------------------
| /
|/ |\_/|
| |q p| /}
| ( 0 )"""\
\ |"-"` |
|| /=\\ |
"'" '"""'</syntaxhighlight>
==== Using external packages: declaring dependences & Using local functions in other files ====

<ol style="list-style-type: decimal;">
<li>'''Declaring dependencies''': The <code>warnings</code> module is actually a built in python package, but suppose we also needed to use <code>numpy</code> in our package. For example, suppose we created another <code>.py</code> file called <code>utils.py</code>, in it we had
<syntaxhighlight lang="python">import numpy as np

def _top_and_bottom(name_length):
length_vec = np.array([8,4]) + name_length
return(length_vec)</syntaxhighlight>
The first thing we’d need to do (so the code works for others) is to tell the package we require the installation of <code>numpy</code> for anyone using our package. We do so by running
<syntaxhighlight lang="bash">poetry add numpy</syntaxhighlight>
This will update our <code>pyproject.toml</code> and add asssociated dependences (there are none for <code>numpy</code>), and will put that note under <code>[tool.poetry.dependencies]</code>.
''Note: as of 8/20/21 this also required restricting my python version (in the <code>pyproject.toml</code> to <code>python = "?=3.8,<3.11"</code>)</li>
<li>'''Internal Functions''': To use our interal function <code>_top_and_bottom</code> we might change the <code>communicate.py</code> file to:
<syntaxhighlight lang="python">import warnings
from .utils import _top_and_bottom

def call_scotty_demo(your_name):
# import warnings
if len(your_name) > 30:
warnings.warn("your name is beyond 30 characters, "+\
"has been truncated")
your_name = your_name[:30]

name_length = len(your_name)

tb_info = _top_and_bottom(name_length)

# say Hiiiiii
hi_string = "\n" +\
"-" * tb_info[0] + "\n" +\
"| Hi " + your_name + "! | \n" +\
"| " + "-" * tb_info[1] + "\n" +\
"| /\n"

...</syntaxhighlight>
Notice that to give <code>call_scotty_demo</code> and all functions in the <code>communicate.py</code> file access to <code>_top_and_bottom</code> we had to import it using <code>from .utils import _top_and_bottom</code>. Additionally, it’s best practice to use add a <code>_</code> at the beginning of a function name if you plan to keep it internal.</li></ol>

=== Step 4: Documenting your functions ===

Python documentation is simplier than R documentation as when you look at the documentation of a python function you just get show a docstring at the head of function. In this section we’ll discuss a bit about the extra support poetry can do so that you can generate a localized website for your documentation (that you could upload to [https:://readthedocs.org readthedocs] later).

For documentation to be easily publishable at a later date we recommend that your docstrings follow the <code>numpydoc</code> format (more details can be found at [https://numpydoc.readthedocs.io/en/latest/format.html#docstring-standard numpydoc.readthedocs]). Using this guideline we can document <code>call_scotty_demo</code> as follows:

<syntaxhighlight lang="python">def call_scotty_demo(your_name):
"""
Greet Scotty, the Scottie Dog

Scotty, the Scottie Dog greets your by name!

Parameters
----------
your_name: str
string of your name (max length 30 characters)

Returns
-------
None
Though scotty appears on your screen and says Hi to you.

Examples
--------
>>> import tartanpy
>>> tartanpy.call_scotty_demo("Andrew Carnegie")

-----------------------
| Hi Andrew Carnegie! |
| -------------------
| /
|/ |\_/|
| |q p| /}
| ( 0 )\"\"\"\\
\\ |\"-\"` |
|| /=\\\\ |
\"'\" '\"\"\"'
"""
...</syntaxhighlight>
The basic structure of the documention is decently straight forward with the title / description of your function first, followed by (1) Parameters header with the parameter name, the type, and the description in the format above (2) Returns header followed by similar structure as Parameters. Adding (3) examples with standard python lines can be helpful as well.

After reloading the package you could see that

<syntaxhighlight lang="python">In [1]: import tartanpy

In [2]: ?tartanpy.call_scotty_demo
Signature: tartanpy.call_scotty_demo(your_name)
Docstring:
Greet Scotty, the Scottie Dog

Scotty, the Scottie Dog greets your by name!

Parameters
----------
your_name: str
string of your name (max length 30 characters)

Returns
-------
None
Though scotty appears on your screen and says Hi to you.

Examples
--------
>>> import tartanpy
>>> tartanpy.call_scotty_demo("Andrew Carnegie")
-----------------------
| Hi Andrew Carnegie! |
| -------------------
| /
|/ |\_/|
| |q p| /}
| ( 0 )"""\
\ |"-"` |
|| /=\\ |
"'" '"""'
File: ..../tartanpy/src/tartanpy/communicate.py
Type: function</syntaxhighlight>
==== Towards "readthedocs"-style html documentation ====

py-pkgs and poetry also provide a quick way to transform your documentation into a local html website so you can quickly look through your documentation or load it to readthedocs later, with information on how to here: [https://py-pkgs.org/06-documentation py-pkgs/06-documentation].

Generally speaking you have to

<ol>
<li> install some additional packages to the development side of your package with

<syntaxhighlight lang="bash">poetry add --dev sphinx nbsphinx ipykernel</syntaxhighlight> </li>

<li>and then go to your docs folder and make the website locally (the <code>--dev</code> parameter makes these packages not tracked for those just wanted to use your package, but required as you continue to develop the package locally).

<syntaxhighlight lang="bash">cd docs
poetry run make html</syntaxhighlight>
and this will create a local webpage (just open <code>docs/index.html</code>) that contains your documentation. If you find the docs website useful maybe bookmark the local file?

</li></ol>

=== Step 5: Testing ===

Testing is critical to making sure your functions do what you expect. There are few ways to write unit tests in python (e.g. <code>unittest</code>, <code>pytest</code> and <code>nose</code>). We recommend using <code>pytest</code> as it requires the least amount of overhead and provides the most details write off the bat.

To use <code>pytest</code> we first to tell <code>poetry</code> that we’ll use <code>pytest</code> in our development with

<syntaxhighlight lang="bash">poetry add --dev pytest</syntaxhighlight>
==== What’s the philosophy behind <code>pytest</code> ====

<ol>
<li>'''Assertions''': <code>pytest</code>’s foundation is defined around checking basic assertion statements like

<syntaxhighlight lang="python">In [1]: assert 1 == 1, "a true statement shouldn't have errored..."
...: assert .3 - .2 == .1, "but sometimes floating points are bad..."</syntaxhighlight>
that return:

<syntaxhighlight lang="python">---------------------------------------------------------------------------
AssertionError Traceback (most recent call last)
<ipython-input-3-0daf5f7e9b9c> in <module>
1 assert 1 == 1, "a true statement shouldn't have errored..."
----> 2 assert .3 - .2 == .1, "but sometimes floating points are bad..."

AssertionError: but sometimes floating points are bad...</syntaxhighlight>
as the second line actual fails (due to some floating point silliness). This simple structure can be combined with standard logical <code>and</code> and <code>or</code> as well as other tools like <code>numpy</code> logic (.e.g <code>np.all()</code>). Notice that following a logical statement with a string nicely associated a statement to be returned if the assertion is not true.
</li>
<li>'''Grouping Assertions''': We can imagine wishing to combine a set of assertions, and we can do this easily with a function wrapper. For example

<syntaxhighlight lang="python">import tartanpy
import pytest # only for warning

def test_call_scotty_demo():
"""
test the basics of call_scotty_demo
"""

assert tartanpy.call_scotty_demo("Oski Bear") is None, \
"the output of call_scotty_demo should be None"

# test warning
with pytest.warns(UserWarning) as record:
tartanpy.call_scotty_demo("Oski Bear, your a wonderful, but slightly creepy Berkeley Bear")

assert len(record) > 0, \
"expect a warning for name_string longer than 30 for call_scotty_demo"

assert .3 - .2 == .1, "but sometimes floating points are bad..."</syntaxhighlight>
</li>
<li>To actually leverage <code>pytest</code> we can put '''A set of grouped tests''' in single file. When building a package your functions will likelyl live in different <code>.py</code> files in the <code>src/<packagename></code> folder. To mirror this structure we put of our tests associated with a single <code>.py</code> file into 1 test file. These files sit in the <code>tests/</code> folder and must start with <code>test</code>.

In these larger files (relative to group functions) need to make sure our functions are brought in, for example at the head we need to put <code>import tartanpy</code> as see in the code block above.

To run these tests you can either do

<syntaxhighlight lang="bash">pytest</syntaxhighlight>
or (making sure you have all the correct packages)

<syntaxhighlight lang="bash">poetry run pytest</syntaxhighlight>
''Note: we actually have some printouts in <code>call_scotty_demo</code>, which are not desirable, so we actually follow a suggestion from [https://stackoverflow.com/questions/8391411/how-to-block-calls-to-print stackoverflow] to suppress the printing (doesn’t mess with the testing), and is seen in the code block below:''

<syntaxhighlight lang="python">import tartanpy
import pytest # only for warning

def test_call_scotty_demo():
"""
test the basics of call_scotty_demo
"""
with open(os.devnull, "w") as f, contextlib.redirect_stdout(f):
assert tartanpy.call_scotty_demo("Oski Bear") is None, \
"the output of call_scotty_demo should be None"

# test warning
with pytest.warns(UserWarning) as record:
tartanpy.call_scotty_demo("Oski Bear, your a wonderful, but slightly creepy Berkeley Bear")

assert len(record) > 0, \
"expect a warning for name_string longer than 30 for call_scotty_demo"

assert .3 - .2 == .1, "but sometimes floating points are bad..."</syntaxhighlight>
If we remove the silly floating point error assession we get the following:

[[File:1200px-Pytest example.png|300px|frame|left|output of pytest]]

And if there are errors <code>pytest</code> tries to be very clear about where in your tests you had an error.
</li></ol>

==== More comments on testing ====

<ol>
<li>'''Hidden and/or not exported functions'''

In our example we have a hidden function <code>_top_and_bottom</code>. Different thoughts exist on whether or not one should test hidden functions, but to do so we have to be a bit tricker since the function isn’t loaded by the package. We can still test it by using the full path to the function, that is:

<syntaxhighlight lang="python">import tartanpy
import numpy as np

def test__top_and_bottom():
"""
test for hidden function _top_and_bottom
"""
option4 = tartanpy.utils._top_and_bottom(len("test"))
option17 = tartanpy.utils._top_and_bottom(len("test_a_longer_one"))

assert np.all(option17 - option4 == 17-4), \
"expect _top_and_bottom values to only differ by length of input"

assert np.all(option17 - 17 == np.array([8,4])), \
"expect that _top_and_bottom adds 8 and 4 to length of input"

</syntaxhighlight>
Additionally, since we started the name off with a <code>_</code>, we could also find that if we did

<syntaxhighlight lang="python">from tartanpy.utils import *</syntaxhighlight>
which should import everything from the <code>util.py</code> file we won’t actually have access to it, whereas

<syntaxhighlight lang="python">from tartanpy.communicate import *</syntaxhighlight>
would get access to <code>call_scotty_demo</code> even if it wasn’t in the <code>__init__.py</code> file.

</li>
<li>'''Numerical instablity'''

We included an example where floating point differnces made <code>.3-.2</code> not equal to <code>.1</code>. To avoid such errors we recommend using tools like

* <code>np.allclose</code> (which allows user to provide relative and absolute tolerance between vectors)
* or do something like <code>np.abs(a-b)<1e-08</code> or <code>np.abs(a-b)/np.abs(b) < 1e-08</code>
</li></ol>

=== Step 6. Sharing / using on other computers ===

You could publish your package to pypi (documented here: [https://py-pkgs.org/07-releasing-versioning py-pkg/07-releasing-versioning]) but if you’re still actively working on a developer version another easy way is to make sure you have a clean copy on github, have the user clone the repo and then run either (when in the top directory of the cloned repo)

<syntaxhighlight lang="bash">pip install .</syntaxhighlight>
or

<syntaxhighlight lang="bash">pip install -e .</syntaxhighlight>
where the first installs the package 1 time, and the second actually installs the package and updates the package if that user was to do more pulls / updates the the package.

=== Step 7. And beyond! ===

In the above example, we still didn’t cover a lot of things, including:

*publishing the package
*publishing the docs
*including data
*CI/CD (which will be provided in another wiki)

See [https://py-pkgs.org py-pkgs] for some good starting blocks for most of these.

Developing python packages

2024-05-24T01:59:05Z

Skipper:

This wiki presents a motivational example of a lot of properties of python packages and how to use them. Please see [[packages motivation]] for motivation about why you should utilize the package framework while doing research. This material captures the basics presented by [https://py-pkgs.org/ py-pkgs], (recently) developed by UBC professors. As such you’ll find that python packages online may have different structures. Still, <code>py-pkgs</code> provides a lot of helpful tools to automate python package development. The <code>py-pkgs</code> project was actually started to mirror the helpful [[Developing_R_packages|package development tools in <code>R</code>]]. Note that this wiki page does provide additional commentary but the process proposed is very similar to <code>py-pkgs</code>’s approach.

== Foundations (what tools will be used) ==

In the package development pipeline that is proposed by <code>py-pkgs</code> we have a few very power packages to help you develop an <code>python</code> package as painlessly as possible.

* <code>poetry</code> help build a python package
* <code>cookiecutter</code> helps utilize pre-made templates (for our package’s files)
* <code>pytest</code> provides clean and standardized tools for testing
* (numpy style code) provides rubrics for documentation

== Building an python package ==

This process is slightly different than the one we demonstrated in the [[Developing_R_packages]] wiki (we set up the structure of the package and then link to github).

=== Step 1: Creating a minimal source package ===

A python (''source'') package just files in a directory, formatted in a specific way.

To create the bare bones python package, we will be using the packages <code>poetry</code> and <code>cookiecutter</code>, so if you haven’t already done so, install those two packages (we’ll be using <code>conda</code>) by typing:

<syntaxhighlight lang="bash">conda install -c conda-forge poetry cookiecutter</syntaxhighlight>
''Note that this line is taken directly from [https://py-pkgs.org/02-setup py-pkgs/02-setup], and it is possible that some day these packages can be installed wit ha simple <code>conda install poetry cookiecutter</code>.''

'''Now''', to actually start setting up your python package we will first use <code>cookiecutter</code> to copy a package structure provided by <code>py-pkgs</code> by running the following in our terminal:

<syntaxhighlight lang="bash">cookiecutter https://github.com/UBC-MDS/cookiecutter-ubc-mds.git</syntaxhighlight>
This will prompt you to answer a set of questions including
<ol>
<li> the author name (can include caps and spaces) </li>
<li> github username </li>
<li> package name (can include caps and spaces) </li>
<li> package slug name (can included underscores but no spaces) </li>
<li> a short package description (can include caps and spaces) </li>
<li> version number (we recommend either 0.0.1 or 0.1.0) </li>
<li> python version 8. open source license (they list the options) </li>
<li> if the package should include github actions (we suggest no for now, and plan to have wikis help with this later) </li>
</ol>

''Note: our package for this example will be named <code>tartanpy</code>'' (an a copy of the "final" product can be found on [https://github.com/benjaminleroy/tartanpy|github].

''Note: the cookiecutter template from <code>py-pkg</code>s (aka the one above) is still under construction - so it might look slightly different in the future.''

This will create a <code>tartanpy</code> folder with the following items:

# Directory <code>src/</code> (where all the function source code goes)
# Directory <code>docs/</code> (where all the pretty looking documentation goes '''- auto generated''')
# Directory <code>tests/</code> (where you’ll store your test, with a placeholder test file)
# <code>pyproject.toml</code> file (a description file about your package, which includes the things you entered when you ran the <code>cookiecutter</code> call)
# <code>LICENSE</code> (booler plate License - don’t really need to deal with this)
# A set of <code>.md</code> files for document things like general info for viewers (<code>README.md</code>), info for contributors and their conduct (<code>CONTRIBUTING.md</code> and <code>CONDUCT.md</code>) and a file to include information on changes you’ve made <code>CHANGELOG.md</code>).

[[File:1200px-Tartanpy folder.png|600px|frame|left|initial directory items]]

==== pyproject.toml file meta-data ====

There are few ways for packages to store their meta-data, and <code>py-pkgs</code>’s path is through the <code>pyproject.toml</code>. Other options include things like <code>setup.py</code> files. If you made a mistake when answering questions via your cookiecutter prompts you can change them by opening up the <code>pyproject.toml</code> file. Our example package’s <code>pyproject.toml</code> file looks like the image below

[[File:1200px-Tartanpy pyproject description.png|600px|frame|left|initialized description file]]

In a <code>pyproject.toml</code> file you’ll notice that each section is headed by a <code>[info]</code> heading. Notice that three of the sections in the image above have <code>tool.poetry</code> headers. This means that the package <code>poetry</code> can update these sections. As we showcase the power of this package you may find that using poetry to update a lot of these fields is more desirable than manually doing it yourself.

=== Step 2: Setting up a Github repo ===

Version control (and discussed in Stat Computing) is a very useful tool to help streamline file management and provide piece of mind. Oftentimes, you will want to have a copy of the repository not just on your personal computer but online (to help with sharing, just in case your computer dies, it’s cool, etc.). As such these steps will walk you through how to set up a github repo for your new package and how to connect that repo to a new python package. This step will mirror thoughts in [https://py-pkgs.org/03-how-to-package-a-python#set-up-remote-version-control py-pkgs/03-how-to-package-a-python].

<ol style="list-style-type: decimal;">
<li>'''In <code>Github</code>''', do the following
<ul>
<li>go to the top-level directory (i.e., <code>github.com/<your user name></code>)</li>
<li>click on “+” at top right, and select “New repository”</li>
<li>name the repository (e.g., “tartan_python”). Note that if you’d like to share your package with others via pypi you’ll need to make your package name unique!</li>
<li>provide a short description of the repository (don’t leave completely blank!)</li>
<li>keep the repository public (as students you have access to free private repos https://education.github.com/pack, but for purposes of this example we will keep the repo public)</li>
<li>'''do not''' initalize with a README, .gitignore, or license (It even says “Skip this step if you’re importing an exisiting repo”!!!)</li>
<li>click on “Create Repository”</li></ul>
</li>
<li>'''…then in your terminal, do the following'''
<ul>
<li>First, initialize your repository with

<syntaxhighlight lang="bash">git init</syntaxhighlight>
then add all <code>cookiecutter</code> files and create your initial commit with <code>bash git add . git commit -m "initial package setup"</code>
</li>
<li>setup connection to github by adding a a remote, your main branch and pushing your first commit with

<syntaxhighlight lang="bash">git remote add origin https://github.com/benjaminleroy/tartanpy.git
git branch -M main
git push -u origin main</syntaxhighlight>
''Note that github provides you with walk throughs that mirror what was recommended above and showcases 2 approaches if you want to use the command line (<code>https</code> and <code>ssh</code>) as well as how you might do it with github desktop. So helpful - thanks github!''</li></ul>

</li></ol>

=== Step 3: Writing a function ===

All of your python code needs to go in <code>.py</code> files in the <code>src/tartanpy/</code> directory. Suppose our first set of functions deal with communicating to the user - then we might make a <code>communicate.py</code> file in <code>src/tartanpy/</code> folder. Inside this file, let’s image we add the following function

<syntaxhighlight lang="python">def call_scotty_demo(your_name):

name_length = len(your_name)

# say Hiiiiii
hi_string = "\n" +\
"-" * (name_length + 8) + "\n" +\
"| Hi " + your_name + "! | \n" +\
"| " + "-" * (name_length + 4) + "\n" +\
"| /\n"

scotty_string = ""+\
" |/ |\\_/| \n" +\
" | |q p| /} \n" +\
" | ( 0 )\"\"\"\\ \n" +\
" \\ |\"-\"` | \n" +\
" || /=\\\\ | \n" +\
" \"'\" '\"\"\"'"

print(hi_string + scotty_string)
</syntaxhighlight>
To get access to this function we can use <code>poetry</code> to locally install our package with the bash command below (that needs to be run in the top folder of our package)

<pre>poetry install</pre>
''Note that without poetry this could be done with a <code>pip install -e .</code> in the same directory and this would actually not require running this line of code every time we update the package. On the other hand - we’ll show the versatility of <code>poetry</code> in the rest of the article.''

To utilize our function we can then open up a interactive python instance (e.g. <code>ipython</code> in your terminal) and then run

<syntaxhighlight lang="python">In [1]: from tartanpy import communicate

In [2]: communicate.call_scotty_demo("Andrew Carnegie")</syntaxhighlight>
which will return:

<syntaxhighlight lang="bash">-----------------------
| Hi Andrew Carnegie! |
| -------------------
| /
|/ |\_/|
| |q p| /}
| ( 0 )"""\
\ |"-"` |
|| /=\\ |
"'" '"""'</syntaxhighlight>
Note that <code>poetry install</code> can be called over and over if you keep updating your function files and will re-load the newest verion of all your functions.

==== Additional notes: ====

===== Understanding <code>__init__.py</code> =====

For <code>python</code> packages you’ll always have an <code>__init__.py</code> file in this directory. We currently have to import the <code>communicate</code> file from <code>tartanpy</code> in order to get access to our amazing function. To make this function come directly with the package, in the <code>__init__.py</code> file we can write

<syntaxhighlight lang="python">from .communicate import call_scotty_demo</syntaxhighlight>
''Note the dot in <code>.communicate</code> says “in this same directory as this current file”, and we don’t have to include the <code>.py</code> file link at the end.'' Additionally This approach '''only''' imports call_scotty_demo to the global package, we could use <code>... import *</code> to import all functions (but this approach is smart if we had helper functions we didn’t really need to share with the end user).*

With the above change, we can then do (the <code>poetry</code> part still needs to be at the top directory):

<syntaxhighlight lang="bash">poetry install
ipython</syntaxhighlight>
and then in the ipython shell:

<syntaxhighlight lang="python">In [1]: import tartanpy

In [2]: tartanpy.call_scotty_demo("Andrew Mellon")</syntaxhighlight>
which will return:

<syntaxhighlight lang="bash">---------------------
| Hi Andrew Mellon! |
| -----------------
| /
|/ |\_/|
| |q p| /}
| ( 0 )"""\
\ |"-"` |
|| /=\\ |
"'" '"""'</syntaxhighlight>
===== Using external packages =====

[You'll want to read the next section as well for a full understanding of dealing with external packages.]

Suppose I wished to truncate really long names I’m greeting but also give the user a warning that I’m going to do so. I’d need to use the <code>warnings</code> package. To use this in our package I’d need to change our <code>communicate.py</code> file to

<syntaxhighlight lang="python">import warnings

def call_scotty_demo(your_name):
if len(your_name) > 30:
warnings.warn("your name is beyond 30 characters, "+\
"has been truncated")
your_name = your_name[:30]

name_length = len(your_name)

# say Hiiiiii
hi_string = "\n" +\
"-" * (name_length + 8) + "\n" +\
"| Hi " + your_name + "! | \n" +\
"| " + "-" * (name_length + 4) + "\n" +\
"| /\n"

scotty_string = ""+\
" |/ |\\_/| \n" +\
" | |q p| /} \n" +\
" | ( 0 )\"\"\"\\ \n" +\
" \\ |\"-\"` | \n" +\
" || /=\\\\ | \n" +\
" \"'\" '\"\"\"'"

print(hi_string + scotty_string)</syntaxhighlight>
Notice that we have to impot the <code>warnings</code> package at the top of our function file.

Now we have the additional funcitonality (after reloading with <code>poetry</code>)

<syntaxhighlight lang="python">In [1]: import tartanpy

In [2]: tartanpy.call_scotty_demo("Oski Bear, your a wonderful, but slightly creepy Berkeley Bear")</syntaxhighlight>
which returns

<syntaxhighlight lang="python">.../tartanpy/src/tartanpy/communicate.py:6: UserWarning: your name is beyond 30 characters, has been truncated
warnings.warn("your name is beyond 30 characters, "+\

--------------------------------------
| Hi Oski Bear, your a wonderful, b! |
| ----------------------------------
| /
|/ |\_/|
| |q p| /}
| ( 0 )"""\
\ |"-"` |
|| /=\\ |
"'" '"""'</syntaxhighlight>
==== Using external packages: declaring dependences & Using local functions in other files ====

<ol style="list-style-type: decimal;">
<li>'''Declaring dependencies''': The <code>warnings</code> module is actually a built in python package, but suppose we also needed to use <code>numpy</code> in our package. For example, suppose we created another <code>.py</code> file called <code>utils.py</code>, in it we had
<syntaxhighlight lang="python">import numpy as np

def _top_and_bottom(name_length):
length_vec = np.array([8,4]) + name_length
return(length_vec)</syntaxhighlight>
The first thing we’d need to do (so the code works for others) is to tell the package we require the installation of <code>numpy</code> for anyone using our package. We do so by running
<syntaxhighlight lang="bash">poetry add numpy</syntaxhighlight>
This will update our <code>pyproject.toml</code> and add asssociated dependences (there are none for <code>numpy</code>), and will put that note under <code>[tool.poetry.dependencies]</code>.
''Note: as of 8/20/21 this also required restricting my python version (in the <code>pyproject.toml</code> to <code>python = "?=3.8,<3.11"</code>)</li>
<li>'''Internal Functions''': To use our interal function <code>_top_and_bottom</code> we might change the <code>communicate.py</code> file to:
<syntaxhighlight lang="python">import warnings
from .utils import _top_and_bottom

def call_scotty_demo(your_name):
# import warnings
if len(your_name) > 30:
warnings.warn("your name is beyond 30 characters, "+\
"has been truncated")
your_name = your_name[:30]

name_length = len(your_name)

tb_info = _top_and_bottom(name_length)

# say Hiiiiii
hi_string = "\n" +\
"-" * tb_info[0] + "\n" +\
"| Hi " + your_name + "! | \n" +\
"| " + "-" * tb_info[1] + "\n" +\
"| /\n"

...</syntaxhighlight>
Notice that to give <code>call_scotty_demo</code> and all functions in the <code>communicate.py</code> file access to <code>_top_and_bottom</code> we had to import it using <code>from .utils import _top_and_bottom</code>. Additionally, it’s best practice to use add a <code>_</code> at the beginning of a function name if you plan to keep it internal.</li></ol>

=== Step 4: Documenting your functions ===

Python documentation is simplier than R documentation as when you look at the documentation of a python function you just get show a docstring at the head of function. In this section we’ll discuss a bit about the extra support poetry can do so that you can generate a localized website for your documentation (that you could upload to [https:://readthedocs.org readthedocs] later).

For documentation to be easily publishable at a later date we recommend that your docstrings follow the <code>numpydoc</code> format (more details can be found at [https://numpydoc.readthedocs.io/en/latest/format.html#docstring-standard numpydoc.readthedocs]). Using this guideline we can document <code>call_scotty_demo</code> as follows:

<syntaxhighlight lang="python">def call_scotty_demo(your_name):
"""
Greet Scotty, the Scottie Dog

Scotty, the Scottie Dog greets your by name!

Parameters
----------
your_name: str
string of your name (max length 30 characters)

Returns
-------
None
Though scotty appears on your screen and says Hi to you.

Examples
--------
>>> import tartanpy
>>> tartanpy.call_scotty_demo("Andrew Carnegie")

-----------------------
| Hi Andrew Carnegie! |
| -------------------
| /
|/ |\_/|
| |q p| /}
| ( 0 )\"\"\"\\
\\ |\"-\"` |
|| /=\\\\ |
\"'\" '\"\"\"'
"""
...</syntaxhighlight>
The basic structure of the documention is decently straight forward with the title / description of your function first, followed by (1) Parameters header with the parameter name, the type, and the description in the format above (2) Returns header followed by similar structure as Parameters. Adding (3) examples with standard python lines can be helpful as well.

After reloading the package you could see that

<syntaxhighlight lang="python">In [1]: import tartanpy

In [2]: ?tartanpy.call_scotty_demo
Signature: tartanpy.call_scotty_demo(your_name)
Docstring:
Greet Scotty, the Scottie Dog

Scotty, the Scottie Dog greets your by name!

Parameters
----------
your_name: str
string of your name (max length 30 characters)

Returns
-------
None
Though scotty appears on your screen and says Hi to you.

Examples
--------
>>> import tartanpy
>>> tartanpy.call_scotty_demo("Andrew Carnegie")
-----------------------
| Hi Andrew Carnegie! |
| -------------------
| /
|/ |\_/|
| |q p| /}
| ( 0 )"""\
\ |"-"` |
|| /=\\ |
"'" '"""'
File: ..../tartanpy/src/tartanpy/communicate.py
Type: function</syntaxhighlight>
==== Towards "readthedocs"-style html documentation ====

py-pkgs and poetry also provide a quick way to transform your documentation into a local html website so you can quickly look through your documentation or load it to readthedocs later, with information on how to here: [https://py-pkgs.org/06-documentation py-pkgs/06-documentation].

Generally speaking you have to

<ol>
<li> install some additional packages to the development side of your package with

<syntaxhighlight lang="bash">poetry add --dev sphinx nbsphinx ipykernel</syntaxhighlight> </li>

<li>and then go to your docs folder and make the website locally (the <code>--dev</code> parameter makes these packages not tracked for those just wanted to use your package, but required as you continue to develop the package locally).

<syntaxhighlight lang="bash">cd docs
poetry run make html</syntaxhighlight>
and this will create a local webpage (just open <code>docs/index.html</code>) that contains your documentation. If you find the docs website useful maybe bookmark the local file?

</li></ol>

=== Step 5: Testing ===

Testing is critical to making sure your functions do what you expect. There are few ways to write unit tests in python (e.g. <code>unittest</code>, <code>pytest</code> and <code>nose</code>). We recommend using <code>pytest</code> as it requires the least amount of overhead and provides the most details write off the bat.

To use <code>pytest</code> we first to tell <code>poetry</code> that we’ll use <code>pytest</code> in our development with

<syntaxhighlight lang="bash">poetry add --dev pytest</syntaxhighlight>
==== What’s the philosophy behind <code>pytest</code> ====

<ol>
<li>'''Assertions''': <code>pytest</code>’s foundation is defined around checking basic assertion statements like

<syntaxhighlight lang="python">In [1]: assert 1 == 1, "a true statement shouldn't have errored..."
...: assert .3 - .2 == .1, "but sometimes floating points are bad..."</syntaxhighlight>
that return:

<syntaxhighlight lang="python">---------------------------------------------------------------------------
AssertionError Traceback (most recent call last)
<ipython-input-3-0daf5f7e9b9c> in <module>
1 assert 1 == 1, "a true statement shouldn't have errored..."
----> 2 assert .3 - .2 == .1, "but sometimes floating points are bad..."

AssertionError: but sometimes floating points are bad...</syntaxhighlight>
as the second line actual fails (due to some floating point silliness). This simple structure can be combined with standard logical <code>and</code> and <code>or</code> as well as other tools like <code>numpy</code> logic (.e.g <code>np.all()</code>). Notice that following a logical statement with a string nicely associated a statement to be returned if the assertion is not true.
</li>
<li>'''Grouping Assertions''': We can imagine wishing to combine a set of assertions, and we can do this easily with a function wrapper. For example

<syntaxhighlight lang="python">import tartanpy
import pytest # only for warning

def test_call_scotty_demo():
"""
test the basics of call_scotty_demo
"""

assert tartanpy.call_scotty_demo("Oski Bear") is None, \
"the output of call_scotty_demo should be None"

# test warning
with pytest.warns(UserWarning) as record:
tartanpy.call_scotty_demo("Oski Bear, your a wonderful, but slightly creepy Berkeley Bear")

assert len(record) > 0, \
"expect a warning for name_string longer than 30 for call_scotty_demo"

assert .3 - .2 == .1, "but sometimes floating points are bad..."</syntaxhighlight>
</li>
<li>To actually leverage <code>pytest</code> we can put '''A set of grouped tests''' in single file. When building a package your functions will likelyl live in different <code>.py</code> files in the <code>src/<packagename></code> folder. To mirror this structure we put of our tests associated with a single <code>.py</code> file into 1 test file. These files sit in the <code>tests/</code> folder and must start with <code>test</code>.

In these larger files (relative to group functions) need to make sure our functions are brought in, for example at the head we need to put <code>import tartanpy</code> as see in the code block above.

To run these tests you can either do

<syntaxhighlight lang="bash">pytest</syntaxhighlight>
or (making sure you have all the correct packages)

<syntaxhighlight lang="bash">poetry run pytest</syntaxhighlight>
''Note: we actually have some printouts in <code>call_scotty_demo</code>, which are not desirable, so we actually follow a suggestion from [https://stackoverflow.com/questions/8391411/how-to-block-calls-to-print stackoverflow] to suppress the printing (doesn’t mess with the testing), and is seen in the code block below:''

<syntaxhighlight lang="python">import tartanpy
import pytest # only for warning

def test_call_scotty_demo():
"""
test the basics of call_scotty_demo
"""
with open(os.devnull, "w") as f, contextlib.redirect_stdout(f):
assert tartanpy.call_scotty_demo("Oski Bear") is None, \
"the output of call_scotty_demo should be None"

# test warning
with pytest.warns(UserWarning) as record:
tartanpy.call_scotty_demo("Oski Bear, your a wonderful, but slightly creepy Berkeley Bear")

assert len(record) > 0, \
"expect a warning for name_string longer than 30 for call_scotty_demo"

assert .3 - .2 == .1, "but sometimes floating points are bad..."</syntaxhighlight>
If we remove the silly floating point error assession we get the following:

[[File:1200px-Pytest example.png|600px|frame|left|output of pytest]]

And if there are errors <code>pytest</code> tries to be very clear about where in your tests you had an error.
</li></ol>

==== More comments on testing ====

<ol>
<li>'''Hidden and/or not exported functions'''

In our example we have a hidden function <code>_top_and_bottom</code>. Different thoughts exist on whether or not one should test hidden functions, but to do so we have to be a bit tricker since the function isn’t loaded by the package. We can still test it by using the full path to the function, that is:

<syntaxhighlight lang="python">import tartanpy
import numpy as np

def test__top_and_bottom():
"""
test for hidden function _top_and_bottom
"""
option4 = tartanpy.utils._top_and_bottom(len("test"))
option17 = tartanpy.utils._top_and_bottom(len("test_a_longer_one"))

assert np.all(option17 - option4 == 17-4), \
"expect _top_and_bottom values to only differ by length of input"

assert np.all(option17 - 17 == np.array([8,4])), \
"expect that _top_and_bottom adds 8 and 4 to length of input"

</syntaxhighlight>
Additionally, since we started the name off with a <code>_</code>, we could also find that if we did

<syntaxhighlight lang="python">from tartanpy.utils import *</syntaxhighlight>
which should import everything from the <code>util.py</code> file we won’t actually have access to it, whereas

<syntaxhighlight lang="python">from tartanpy.communicate import *</syntaxhighlight>
would get access to <code>call_scotty_demo</code> even if it wasn’t in the <code>__init__.py</code> file.

</li>
<li>'''Numerical instablity'''

We included an example where floating point differnces made <code>.3-.2</code> not equal to <code>.1</code>. To avoid such errors we recommend using tools like

* <code>np.allclose</code> (which allows user to provide relative and absolute tolerance between vectors)
* or do something like <code>np.abs(a-b)<1e-08</code> or <code>np.abs(a-b)/np.abs(b) < 1e-08</code>
</li></ol>

=== Step 6. Sharing / using on other computers ===

You could publish your package to pypi (documented here: [https://py-pkgs.org/07-releasing-versioning py-pkg/07-releasing-versioning]) but if you’re still actively working on a developer version another easy way is to make sure you have a clean copy on github, have the user clone the repo and then run either (when in the top directory of the cloned repo)

<syntaxhighlight lang="bash">pip install .</syntaxhighlight>
or

<syntaxhighlight lang="bash">pip install -e .</syntaxhighlight>
where the first installs the package 1 time, and the second actually installs the package and updates the package if that user was to do more pulls / updates the the package.

=== Step 7. And beyond! ===

In the above example, we still didn’t cover a lot of things, including:

*publishing the package
*publishing the docs
*including data
*CI/CD (which will be provided in another wiki)

See [https://py-pkgs.org py-pkgs] for some good starting blocks for most of these.

File:1200px-Pytest example.png

2024-05-24T01:58:39Z

Skipper:

output of pytest

File:1200px-Tartanpy pyproject description.png

2024-05-24T01:54:09Z

Skipper:

initialized description file

File:1200px-Tartanpy folder.png

2024-05-24T01:53:18Z

Skipper:

initial directory items

Developing R packages

2024-05-24T01:49:42Z

Skipper:

This wiki presents a motivational example of a lot of properties of R packages and how to use them. Please see [[packages motivation]] for motivation about why you should utilize the package framework while doing research. The material in this page captures thoughts from lots of people and is a rewriting of a lecture Ben LeRoy presented while teaching 36-350 in the summer of 2019. Additionally, there are a lot of useful resources on the web to help you make an <code>R</code> package including:

* [http://r-pkgs.had.co.nz R Packages]. ~Hadley Wickham
* [https://link.springer.com/book/10.1007/978-0-387-75936-4 Software for Data Analysis: Programming with R.] ~John Chambers
* [https://cran.r-project.org/doc/manuals/r-release/R-exts.html Writing R Extensions] (the official documentation)

What’s nice about the <code>R</code> programming language how many easy-to-use resources there are to help you develop your own package. The following quote captures a notation that if you’re going to be working with R for long enough, package-driven development is going to happen - so why don’t you start now?

<blockquote>“… from user to programmer to contributor, in the gradual progress that R encourages.” ~ John Chambers (Software for Data Analysis: Programming with R)
</blockquote>

== Foundations (what tools will be used) ==

Currently, there are few very powerful packages to help you develop an <code>R</code> package as painlessly as possible.

* <code>devtools</code> and <code>usethis</code> packages provide a set of functions to automate higher level package details and repeative package development demands
* <code>Roxygen2</code> helps rapid and standardized documentation for your package
* <code>testthat</code> provides clean and standardized tools for testing

== Building an R package ==

=== Step 0: Setting up a Github repo and a Rstudio project directory ===

Version control (and discussed in Stat Computing) is a very useful tool to help streamline file management and provide piece of mind. Oftentimes, you will want to have a copy of the repository not just on your personal computer but online (to help with sharing, just in case your computer dies, it’s cool, etc.). As such these steps will walk you through how to set up a github repo for your new package and how to connect that repo to a new project in R studio.

<ol>
<li>'''In <code>GitHub</code>''', do the following:
<ul>
<li>go to the top-level directory (i.e., <code>github.com/<your user name></code>) </li>
<li>click on “+” at top right, and select “New repository” </li>
<li>name the repository (e.g., “tartan”) </li>
<li>provide a short description of the repository (don’t leave completely blank!) </li>
<li>keep the repository public (as students you have access to free private repos https://education.github.com/pack, but for purposes of this example we will keep the repo public) </li>
<li>click on “Initialize this repository with a README” and select the R option in “Add .gitignore”… there is no need to “Add a license” </li>
<li>click on “Create Repository” </li>
</ul></li>
<li>'''…then in RStudio''', do the following:
<ul>
<li>click on <code>File > New Project...</code> </li>
<li>click on “Version Control”, then on “Git” </li>
<li>provide the ''full'' address for the “Repository URL” (including the <code>https</code>, etc.; by default, this will provide the name of your local repository) </li>
<li>make sure “Create project as subdirectory of:” points to where you want to point it </li>
<li>click on “Create Project” (in this example we will be calling our package <code>tartan</code>) </li>
</ul>
</li>
</ol>

=== Step 1: Creating a minimal source package ===

An R (''source'') package just files in a directory, formatted in a specific way.

To create the bare bones R package, just type in your Rstudio console:

<syntaxhighlight lang="r">usethis::create_package("tartan")</syntaxhighlight>
<syntaxhighlight lang="r"># output of the above command
> usethis::create_package("tartan")
✔ Setting active project to './tartan'
✔ Creating 'R/'
✔ Creating 'man/'
✔ Writing 'DESCRIPTION'
✔ Writing 'NAMESPACE'
✔ Changing working directory to './tartan'</syntaxhighlight>
* if in the <code>tartan</code> project, may need to do <code>setwd("../")</code>, run the above line and then do <code>setwd("tartan")</code>.

This will create a <code>tartan</code> folder with the following items:

# Directory <code>R/</code> (where all the function source code goes)
# Directory <code>man/</code> (where all the <code>Roxygen</code> created documentation goes '''- auto generated''')
# <code>NAMESPACE</code> file (information about what functions you’re providing the user '''- auto generated''')
# <code>DESCRIPTION</code> file (information about your package)
# <code>tartan.Rproj</code> file (if using RStudio - just R project information '''- auto generated''')

and anything else you included from intializing your git repo (<code>.gitignore</code>, <code>LICENSE</code>, etc)

[[File:1200px-Tartan folder.png|600px|frameless|left|initial directory items]]

==== Description file meta-data ====

The first thing you should do is update a little bit of the meta-data in the <code>DESCRIPTION</code> file, an example of which is seen below. You don’t need to add lots, but you’ll probably want to at least add your name and a title.

[[File:Tartan description.png|600px|frameless|left|initialized description file]]

=== Step 2: Writing a function ===

All of your R code needs to go in <code>.R</code> files in the <code>R/</code> directory. You can create these files yourself, but you can also automate this with

<syntaxhighlight lang="r">usethis::use_r("welcome") # opens up file (creates if need be)</syntaxhighlight>
With the previous line of code creating an empty <code>R/welcome.R</code> file. Inside this file, let’s image we add the following function

<syntaxhighlight lang="r">call_scotty_demo <- function(your_name) {
if (nchar(your_name) > 30){
warning("your name is beyond 30 characters, and has been truncated")
your_name <- substr(your_name, 1, 30)
}

name_length <- nchar(your_name)

# Hiiiii
cat(paste("\n",
paste0(c("", rep("-", name_length + 8), "\n"), collapse = ""),
paste0(c("| Hi ", your_name, "! | \n"), collapse=""),
paste0(c("| ", rep("-", name_length + 4), "\n"), collapse = ""),
paste0(c("| /\n")))
)

# from Scotty.
cat(paste0(
" |/ |\\_/| \n",
" | |q p| /} \n",
" | ( 0 )\"\"\"\\ \n",
" \\ |\"-\"` | \n",
" || /=\\\\ | \n",
" \"'\" '\"\"\"'"))

}</syntaxhighlight>
To get access to this function we can locally load the package into memory (a weaker install than calling <code>library()</code>) by running:

<syntaxhighlight lang="r">devtools::load_all()</syntaxhighlight>
Note that <code>load_all()</code> can be called over and over if you keep updating your function files and will re-load the newest verion of all your functions.

==== Additional notes ====

Unlike writing code in an <code>.Rmd</code> or a <code>.R</code> file, we deal with library dependencies slightly differently. If you know functions need tools from a certain library we put that library in the <code>Imports:</code> section of your <code>DESCRIPTION</code> file. Additional to use functions from outside libraries in our functions we have to write them like <code>package_name::function_name</code>. For example in our function <code>call_scotty_demo</code> if we wanted to use <code>stringr</code>’s <code>str_sub</code> function we’d have to have our code look like:

<syntaxhighlight lang="r">call_scotty_demo <- function(your_name) {
if (nchar(your_name) > 30){
warning("your name is beyond 30 characters, and has been truncated")
your_name <- stringr::str_sub(your_name, 1, 30)
}

name_length <- nchar(your_name)

# Hiiiii
cat(paste("\n",
paste0(c("", rep("-", name_length + 8), "\n"), collapse = ""),
paste0(c("| Hi ", your_name, "! | \n"), collapse=""),
paste0(c("| ", rep("-", name_length + 4), "\n"), collapse = ""),
paste0(c("| /\n")))
)

# from Scotty.
cat(paste0(
" |/ |\\_/| \n",
" | |q p| /} \n",
" | ( 0 )\"\"\"\\ \n",
" \\ |\"-\"` | \n",
" || /=\\\\ | \n",
" \"'\" '\"\"\"'"))

}</syntaxhighlight>
and our <code>DESCRIPTION</code> file would need lines like

<syntaxhighlight lang="r">Imports:
stringr</syntaxhighlight>
=== Step 3: Documenting your functions ===

With <code>devtools</code> and a package it’s actually really easy to create local documentation for your own functions so that <code>?function_name</code> pops up documentation in the same way <code>?sum</code> does in Rstudio.

The easiest way to document your code is using <code>Roxygen2</code>’s helper tools. To start, make sure you’ve downloaded the package:

<syntaxhighlight lang="r">install.packages("roxygen2")
library(roxygen2)</syntaxhighlight>
Then at the first line of a function you’ve written (when you’re in Rstudio) use the short-cut <code>Command + Option + Shift + R</code> (Mac) or <code>Ctrl+ Alt + Shift + R</code> (Windows), this will auto-generate a documentation structure to fill in.

For example, if we had the function from above used the shortcut, we would then have:

<syntaxhighlight lang="r">#' Title
#'
#' @param your_name
#'
#' @return
#' @export
#'
#' @examples
call_scotty_demo <- function(your_name) {
...
}</syntaxhighlight>
which we could fill in with:

<syntaxhighlight lang="r">#' Meet Scotty, the Scottie Dog
#'
#' @param your_name string of your name (max length 30 characters)
#'
#' @return NULL. Though scotty appears on your screen and says Hi to you.
#' @export
#'
#' @examples
#' call_scotty_demo("Andrew Carnegie")
call_scotty_demo <- function(your_name) {
...
}</syntaxhighlight>
The basic structure of the documentation is decently straightforward (you can remove <code>@export</code> if you don’t want the function to export, and you can remove <code>@examples</code> if you don’t have any examples), but detailed guidance, documentation [https://r-pkgs.org/man.html?q=roxygen#man here] can be super helpful (with information on linking functions and other packages, adding math equations, code text, etc.)

'''Now''' to actually create a documentation file, we run

<syntaxhighlight lang="r">devtools::document()</syntaxhighlight>
The first time we do this we’ll create <code>man/</code> folder if one doesn’t always exist and also create / update a corresponding file named <code>man/call_scotty_demo.Rd</code> file that R uses to generate documentation. Additionally, the <code>document()</code> function also updates the <code>NAMESPACE</code> file so when you load your functions you get the desired documenation (need to have <code>@export</code> in the docstring).

The <code>man/call_scotty_demo.Rd</code> document file looks like the following. Notice how it says “do not edit by hand”.

<code>{r eval=F} % Generated by roxygen2: do not edit by hand % Please edit documentation in R/welcome.R \name{call_scotty_demo} \alias{call_scotty_demo} \title{Meet Scotty, the Scottie Dog} \usage{ call_scotty_demo(your_name) } \arguments{ \item{your_name}{string of your name (max length 30 characters)} } \value{ \code{NULL}. Though scotty appears on your screen and says Hi to you. } \description{ Meet Scotty, the Scottie Dog } \examples{ call_scotty_demo("Andrew Carnegie") }</code>

To actually view your documentation files we need to re-load the package and then use the standard <code>?</code> to get the help file:

<syntaxhighlight lang="r">devtools::load_all()
?call_scotty_demo</syntaxhighlight>
[[File:Call scotty demo doc.png|600px|frameless|left|example of help file]]

=== Step 4: Testing ===

Testing is critical to making sure your functions do what you expect. The easiest way to get started with testing is the <code>testthat</code> package, which also integrates with <code>devtools</code>. To start testing using <code>testthat</code> in your new R package do the following:

<code>{r eval = F} install.packages("testthat") usethis::use_testthat()</code>

This will set up the <code>tests/testthat</code> directory where we can store tests.

==== What’s the philosophy behind <code>testthat</code> ====

<code>testthat</code> functionality can be see as tools to do a range of needs, from low level testing all the way up to overall testing of all functions in a package.

<ol>
<li>On the low level, <code>testthat</code> provides functions to test your '''expectations'''. The most fundamental unit of testing is this low level. <code>testthat</code>’s <code>expect_...</code> functions provide the building blocks for testing. They describe what result is expected for your various computations and we can use these to make sure our functions are giving the expected output. For example:

<syntaxhighlight lang="r">library(testthat)
expect_equal(1, 1)
expect_equal(1, 2)</syntaxhighlight>
Returns:

<syntaxhighlight lang="r">## Error: 1 not equal to 2.
## 1/1 mismatches
## [1] 1 - 2 == -1</syntaxhighlight>
</li>
<li>'''Grouping expectations''' is the next level. We can group a set of expectations to examine more complex functionality with the wrapper <code>test_that</code> which takes in a string describing what you’re examining, and then a set of <code>expect_...</code>s that are wrapping in <code>{}</code>. For example:

<syntaxhighlight lang="r">test_that(paste("calling scotty works well (returns null",
"and provides warning nchar > 30)"), {
expect_equal(call_scotty_demo("Oski Bear"), NULL)
expect_warning(call_scotty_demo("Oski Bear, your a wonderful, but slightly creepy Berkeley Bear"))
expect_equal(1, 2)
}</syntaxhighlight>
Returns:

<syntaxhighlight lang="r">##
## -----------------
## | Hi Oski Bear! |
## | -------------
## | /
## |/ |\_/|
## | |q p| /}
## | ( 0 )"""\
## \ |"-"` |
## || /=\\ |
## "'" '"""'
## --------------------------------------
## | Hi Oski Bear, your a wonderful, b! |
## | ----------------------------------
## | /
## |/ |\_/|
## | |q p| /}
## | ( 0 )"""\
## \ |"-"` |
## || /=\\ |
## "'" '"""'

## Error: Test failed: 'calling scotty works well (returns null and provides warning nchar > 30)'
## * 1 not equal to 2.
## 1/1 mismatches
## [1] 1 - 2 == -1</syntaxhighlight>
</li>
<li>'''A set of grouped tests''' in single file. When building a package your functions will likely like in different <code>.R</code> files in the <code>R/</code> folder. To mirror this structure we put of our tests associated with a single <code>.R</code> file into 1 test file. These files sit in the <code>tests/testthat/</code> folder and must start with <code>test</code>. <code>devtools</code> and <code>usethis</code> provide a way to create these test in an automated fashion. For example, if we wanted to create a file <code>tests/testthat/test-welcome.R</code>, we could call

<syntaxhighlight lang="r">usethis::use_test("welcome")</syntaxhighlight>
which provides the message:

<syntaxhighlight lang="r">✔ Setting active project to './tartan'
✔ Adding 'testthat' to Suggests field in DESCRIPTION
✔ Creating 'tests/testthat/'
✔ Writing 'tests/testthat.R'
✔ Writing 'tests/testthat/test-welcome.R'
● Modify 'tests/testthat/test-welcome.R'</syntaxhighlight>
In these larger files (relative to group functions) we include group expectations as well as context information for the overall file using the <code>context()</code> function. And example can be see below:

<syntaxhighlight lang="r">context("test welcome from scotty")

test_that(paste("calling scotty works well (returns null",
"and provides warning nchar > 30)"), {
expect_equal(call_scotty_demo("Oski Bear"), NULL)
expect_warning(call_scotty_demo("Oski Bear, your a wonderful, but slightly creepy Berkeley Bear"))
expect_equal(1, 2)
})</syntaxhighlight>
Once you have a test file in your package all you need to do is run

<syntaxhighlight lang="r">devtools::test()</syntaxhighlight>
and R will run all your associated test, and will return output like seen in the following image:

[[File:Devtool test example.png|600px|frame|left|output of testthat]]
</li>
</ol>

=== Step 5. Checking the full package ===

To understanding if your overall package is clean and able to be shared we can run

<pre>devtools::check()</pre>

This checks all your dependencies, syntax, and more that may have slipped through the cracks. Often times this is only necessary when you’re looking to share your package with others, but is a good step to include so you don’t have to change a lot of things right before sharing.

=== Step 6: Syncronizing with Github ===

As always, once you’ve completed a clean block of code you should push it to Github. When your code is on github other folks can download the package in a similar fashion to the standard <code>install.packages(...)</code>,

For example, to install the R package we just made we could use:

<code>{r eval = T, message=F, warning=F} devtools::install_github(repo = "benjaminleroy/tartan", force = F) # so making the sides does do it a ton of times library(tartan) #?call_scotty_demo call_scotty_demo("Big Bird")</code>

=== Step 7: And beyond! ===

In the above example we still didn’t cover a lot of things, including:

<ul>
<li> Including data in your package (that can be [http://r-pkgs.had.co.nz/data.html documented] too)
<code>r # suppose that I have a tarans dataset that I created - than I can save it this way usethis::use_data(tartans) # and document at usethis::use_r("data")</code> </li>
<li> Vignettes (long form documentation, knitr) </li>
<li> Compiled code (C, C++, Fortran…) </li>
<li> Creating a CRAN ready package </li>
<li> See [https://usethis.r-lib.org/reference/index.html <code>usethis</code>] to see all the package development steps that <code>usethis</code> automates. </li>
</ul>

File:Devtool test example.png

2024-05-24T01:49:27Z

Skipper:

output of testthat

Developing R packages

2024-05-24T01:48:06Z

Skipper:

This wiki presents a motivational example of a lot of properties of R packages and how to use them. Please see [[packages motivation]] for motivation about why you should utilize the package framework while doing research. The material in this page captures thoughts from lots of people and is a rewriting of a lecture Ben LeRoy presented while teaching 36-350 in the summer of 2019. Additionally, there are a lot of useful resources on the web to help you make an <code>R</code> package including:

* [http://r-pkgs.had.co.nz R Packages]. ~Hadley Wickham
* [https://link.springer.com/book/10.1007/978-0-387-75936-4 Software for Data Analysis: Programming with R.] ~John Chambers
* [https://cran.r-project.org/doc/manuals/r-release/R-exts.html Writing R Extensions] (the official documentation)

What’s nice about the <code>R</code> programming language how many easy-to-use resources there are to help you develop your own package. The following quote captures a notation that if you’re going to be working with R for long enough, package-driven development is going to happen - so why don’t you start now?

<blockquote>“… from user to programmer to contributor, in the gradual progress that R encourages.” ~ John Chambers (Software for Data Analysis: Programming with R)
</blockquote>

== Foundations (what tools will be used) ==

Currently, there are few very powerful packages to help you develop an <code>R</code> package as painlessly as possible.

* <code>devtools</code> and <code>usethis</code> packages provide a set of functions to automate higher level package details and repeative package development demands
* <code>Roxygen2</code> helps rapid and standardized documentation for your package
* <code>testthat</code> provides clean and standardized tools for testing

== Building an R package ==

=== Step 0: Setting up a Github repo and a Rstudio project directory ===

Version control (and discussed in Stat Computing) is a very useful tool to help streamline file management and provide piece of mind. Oftentimes, you will want to have a copy of the repository not just on your personal computer but online (to help with sharing, just in case your computer dies, it’s cool, etc.). As such these steps will walk you through how to set up a github repo for your new package and how to connect that repo to a new project in R studio.

<ol>
<li>'''In <code>GitHub</code>''', do the following:
<ul>
<li>go to the top-level directory (i.e., <code>github.com/<your user name></code>) </li>
<li>click on “+” at top right, and select “New repository” </li>
<li>name the repository (e.g., “tartan”) </li>
<li>provide a short description of the repository (don’t leave completely blank!) </li>
<li>keep the repository public (as students you have access to free private repos https://education.github.com/pack, but for purposes of this example we will keep the repo public) </li>
<li>click on “Initialize this repository with a README” and select the R option in “Add .gitignore”… there is no need to “Add a license” </li>
<li>click on “Create Repository” </li>
</ul></li>
<li>'''…then in RStudio''', do the following:
<ul>
<li>click on <code>File > New Project...</code> </li>
<li>click on “Version Control”, then on “Git” </li>
<li>provide the ''full'' address for the “Repository URL” (including the <code>https</code>, etc.; by default, this will provide the name of your local repository) </li>
<li>make sure “Create project as subdirectory of:” points to where you want to point it </li>
<li>click on “Create Project” (in this example we will be calling our package <code>tartan</code>) </li>
</ul>
</li>
</ol>

=== Step 1: Creating a minimal source package ===

An R (''source'') package just files in a directory, formatted in a specific way.

To create the bare bones R package, just type in your Rstudio console:

<syntaxhighlight lang="r">usethis::create_package("tartan")</syntaxhighlight>
<syntaxhighlight lang="r"># output of the above command
> usethis::create_package("tartan")
✔ Setting active project to './tartan'
✔ Creating 'R/'
✔ Creating 'man/'
✔ Writing 'DESCRIPTION'
✔ Writing 'NAMESPACE'
✔ Changing working directory to './tartan'</syntaxhighlight>
* if in the <code>tartan</code> project, may need to do <code>setwd("../")</code>, run the above line and then do <code>setwd("tartan")</code>.

This will create a <code>tartan</code> folder with the following items:

# Directory <code>R/</code> (where all the function source code goes)
# Directory <code>man/</code> (where all the <code>Roxygen</code> created documentation goes '''- auto generated''')
# <code>NAMESPACE</code> file (information about what functions you’re providing the user '''- auto generated''')
# <code>DESCRIPTION</code> file (information about your package)
# <code>tartan.Rproj</code> file (if using RStudio - just R project information '''- auto generated''')

and anything else you included from intializing your git repo (<code>.gitignore</code>, <code>LICENSE</code>, etc)

[[File:1200px-Tartan folder.png|600px|frameless|left|initial directory items]]

==== Description file meta-data ====

The first thing you should do is update a little bit of the meta-data in the <code>DESCRIPTION</code> file, an example of which is seen below. You don’t need to add lots, but you’ll probably want to at least add your name and a title.

[[File:Tartan description.png|600px|frameless|left|initialized description file]]

=== Step 2: Writing a function ===

All of your R code needs to go in <code>.R</code> files in the <code>R/</code> directory. You can create these files yourself, but you can also automate this with

<syntaxhighlight lang="r">usethis::use_r("welcome") # opens up file (creates if need be)</syntaxhighlight>
With the previous line of code creating an empty <code>R/welcome.R</code> file. Inside this file, let’s image we add the following function

<syntaxhighlight lang="r">call_scotty_demo <- function(your_name) {
if (nchar(your_name) > 30){
warning("your name is beyond 30 characters, and has been truncated")
your_name <- substr(your_name, 1, 30)
}

name_length <- nchar(your_name)

# Hiiiii
cat(paste("\n",
paste0(c("", rep("-", name_length + 8), "\n"), collapse = ""),
paste0(c("| Hi ", your_name, "! | \n"), collapse=""),
paste0(c("| ", rep("-", name_length + 4), "\n"), collapse = ""),
paste0(c("| /\n")))
)

# from Scotty.
cat(paste0(
" |/ |\\_/| \n",
" | |q p| /} \n",
" | ( 0 )\"\"\"\\ \n",
" \\ |\"-\"` | \n",
" || /=\\\\ | \n",
" \"'\" '\"\"\"'"))

}</syntaxhighlight>
To get access to this function we can locally load the package into memory (a weaker install than calling <code>library()</code>) by running:

<syntaxhighlight lang="r">devtools::load_all()</syntaxhighlight>
Note that <code>load_all()</code> can be called over and over if you keep updating your function files and will re-load the newest verion of all your functions.

==== Additional notes ====

Unlike writing code in an <code>.Rmd</code> or a <code>.R</code> file, we deal with library dependencies slightly differently. If you know functions need tools from a certain library we put that library in the <code>Imports:</code> section of your <code>DESCRIPTION</code> file. Additional to use functions from outside libraries in our functions we have to write them like <code>package_name::function_name</code>. For example in our function <code>call_scotty_demo</code> if we wanted to use <code>stringr</code>’s <code>str_sub</code> function we’d have to have our code look like:

<syntaxhighlight lang="r">call_scotty_demo <- function(your_name) {
if (nchar(your_name) > 30){
warning("your name is beyond 30 characters, and has been truncated")
your_name <- stringr::str_sub(your_name, 1, 30)
}

name_length <- nchar(your_name)

# Hiiiii
cat(paste("\n",
paste0(c("", rep("-", name_length + 8), "\n"), collapse = ""),
paste0(c("| Hi ", your_name, "! | \n"), collapse=""),
paste0(c("| ", rep("-", name_length + 4), "\n"), collapse = ""),
paste0(c("| /\n")))
)

# from Scotty.
cat(paste0(
" |/ |\\_/| \n",
" | |q p| /} \n",
" | ( 0 )\"\"\"\\ \n",
" \\ |\"-\"` | \n",
" || /=\\\\ | \n",
" \"'\" '\"\"\"'"))

}</syntaxhighlight>
and our <code>DESCRIPTION</code> file would need lines like

<syntaxhighlight lang="r">Imports:
stringr</syntaxhighlight>
=== Step 3: Documenting your functions ===

With <code>devtools</code> and a package it’s actually really easy to create local documentation for your own functions so that <code>?function_name</code> pops up documentation in the same way <code>?sum</code> does in Rstudio.

The easiest way to document your code is using <code>Roxygen2</code>’s helper tools. To start, make sure you’ve downloaded the package:

<syntaxhighlight lang="r">install.packages("roxygen2")
library(roxygen2)</syntaxhighlight>
Then at the first line of a function you’ve written (when you’re in Rstudio) use the short-cut <code>Command + Option + Shift + R</code> (Mac) or <code>Ctrl+ Alt + Shift + R</code> (Windows), this will auto-generate a documentation structure to fill in.

For example, if we had the function from above used the shortcut, we would then have:

<syntaxhighlight lang="r">#' Title
#'
#' @param your_name
#'
#' @return
#' @export
#'
#' @examples
call_scotty_demo <- function(your_name) {
...
}</syntaxhighlight>
which we could fill in with:

<syntaxhighlight lang="r">#' Meet Scotty, the Scottie Dog
#'
#' @param your_name string of your name (max length 30 characters)
#'
#' @return NULL. Though scotty appears on your screen and says Hi to you.
#' @export
#'
#' @examples
#' call_scotty_demo("Andrew Carnegie")
call_scotty_demo <- function(your_name) {
...
}</syntaxhighlight>
The basic structure of the documentation is decently straightforward (you can remove <code>@export</code> if you don’t want the function to export, and you can remove <code>@examples</code> if you don’t have any examples), but detailed guidance, documentation [https://r-pkgs.org/man.html?q=roxygen#man here] can be super helpful (with information on linking functions and other packages, adding math equations, code text, etc.)

'''Now''' to actually create a documentation file, we run

<syntaxhighlight lang="r">devtools::document()</syntaxhighlight>
The first time we do this we’ll create <code>man/</code> folder if one doesn’t always exist and also create / update a corresponding file named <code>man/call_scotty_demo.Rd</code> file that R uses to generate documentation. Additionally, the <code>document()</code> function also updates the <code>NAMESPACE</code> file so when you load your functions you get the desired documenation (need to have <code>@export</code> in the docstring).

The <code>man/call_scotty_demo.Rd</code> document file looks like the following. Notice how it says “do not edit by hand”.

<code>{r eval=F} % Generated by roxygen2: do not edit by hand % Please edit documentation in R/welcome.R \name{call_scotty_demo} \alias{call_scotty_demo} \title{Meet Scotty, the Scottie Dog} \usage{ call_scotty_demo(your_name) } \arguments{ \item{your_name}{string of your name (max length 30 characters)} } \value{ \code{NULL}. Though scotty appears on your screen and says Hi to you. } \description{ Meet Scotty, the Scottie Dog } \examples{ call_scotty_demo("Andrew Carnegie") }</code>

To actually view your documentation files we need to re-load the package and then use the standard <code>?</code> to get the help file:

<syntaxhighlight lang="r">devtools::load_all()
?call_scotty_demo</syntaxhighlight>
[[File:Call scotty demo doc.png|600px|frameless|left|example of help file]]

=== Step 4: Testing ===

Testing is critical to making sure your functions do what you expect. The easiest way to get started with testing is the <code>testthat</code> package, which also integrates with <code>devtools</code>. To start testing using <code>testthat</code> in your new R package do the following:

<code>{r eval = F} install.packages("testthat") usethis::use_testthat()</code>

This will set up the <code>tests/testthat</code> directory where we can store tests.

==== What’s the philosophy behind <code>testthat</code> ====

<code>testthat</code> functionality can be see as tools to do a range of needs, from low level testing all the way up to overall testing of all functions in a package.

<ol>
<li>On the low level, <code>testthat</code> provides functions to test your '''expectations'''. The most fundamental unit of testing is this low level. <code>testthat</code>’s <code>expect_...</code> functions provide the building blocks for testing. They describe what result is expected for your various computations and we can use these to make sure our functions are giving the expected output. For example:

<syntaxhighlight lang="r">library(testthat)
expect_equal(1, 1)
expect_equal(1, 2)</syntaxhighlight>
Returns:

<syntaxhighlight lang="r">## Error: 1 not equal to 2.
## 1/1 mismatches
## [1] 1 - 2 == -1</syntaxhighlight>
</li>
<li>'''Grouping expectations''' is the next level. We can group a set of expectations to examine more complex functionality with the wrapper <code>test_that</code> which takes in a string describing what you’re examining, and then a set of <code>expect_...</code>s that are wrapping in <code>{}</code>. For example:

<syntaxhighlight lang="r">test_that(paste("calling scotty works well (returns null",
"and provides warning nchar > 30)"), {
expect_equal(call_scotty_demo("Oski Bear"), NULL)
expect_warning(call_scotty_demo("Oski Bear, your a wonderful, but slightly creepy Berkeley Bear"))
expect_equal(1, 2)
}</syntaxhighlight>
Returns:

<syntaxhighlight lang="r">##
## -----------------
## | Hi Oski Bear! |
## | -------------
## | /
## |/ |\_/|
## | |q p| /}
## | ( 0 )"""\
## \ |"-"` |
## || /=\\ |
## "'" '"""'
## --------------------------------------
## | Hi Oski Bear, your a wonderful, b! |
## | ----------------------------------
## | /
## |/ |\_/|
## | |q p| /}
## | ( 0 )"""\
## \ |"-"` |
## || /=\\ |
## "'" '"""'

## Error: Test failed: 'calling scotty works well (returns null and provides warning nchar > 30)'
## * 1 not equal to 2.
## 1/1 mismatches
## [1] 1 - 2 == -1</syntaxhighlight>
</li>
<li>'''A set of grouped tests''' in single file. When building a package your functions will likely like in different <code>.R</code> files in the <code>R/</code> folder. To mirror this structure we put of our tests associated with a single <code>.R</code> file into 1 test file. These files sit in the <code>tests/testthat/</code> folder and must start with <code>test</code>. <code>devtools</code> and <code>usethis</code> provide a way to create these test in an automated fashion. For example, if we wanted to create a file <code>tests/testthat/test-welcome.R</code>, we could call

<syntaxhighlight lang="r">usethis::use_test("welcome")</syntaxhighlight>
which provides the message:

<syntaxhighlight lang="r">✔ Setting active project to './tartan'
✔ Adding 'testthat' to Suggests field in DESCRIPTION
✔ Creating 'tests/testthat/'
✔ Writing 'tests/testthat.R'
✔ Writing 'tests/testthat/test-welcome.R'
● Modify 'tests/testthat/test-welcome.R'</syntaxhighlight>
In these larger files (relative to group functions) we include group expectations as well as context information for the overall file using the <code>context()</code> function. And example can be see below:

<syntaxhighlight lang="r">context("test welcome from scotty")

test_that(paste("calling scotty works well (returns null",
"and provides warning nchar > 30)"), {
expect_equal(call_scotty_demo("Oski Bear"), NULL)
expect_warning(call_scotty_demo("Oski Bear, your a wonderful, but slightly creepy Berkeley Bear"))
expect_equal(1, 2)
})</syntaxhighlight>
Once you have a test file in your package all you need to do is run

<syntaxhighlight lang="r">devtools::test()</syntaxhighlight>
and R will run all your associated test, and will return output like seen in the following image:

[[File:devtool_test_example.png|frameless|600px|alt=output of testthat|output of testthat]]
</li>
</ol>

=== Step 5. Checking the full package ===

To understanding if your overall package is clean and able to be shared we can run

<pre>devtools::check()</pre>

This checks all your dependencies, syntax, and more that may have slipped through the cracks. Often times this is only necessary when you’re looking to share your package with others, but is a good step to include so you don’t have to change a lot of things right before sharing.

=== Step 6: Syncronizing with Github ===

As always, once you’ve completed a clean block of code you should push it to Github. When your code is on github other folks can download the package in a similar fashion to the standard <code>install.packages(...)</code>,

For example, to install the R package we just made we could use:

<code>{r eval = T, message=F, warning=F} devtools::install_github(repo = "benjaminleroy/tartan", force = F) # so making the sides does do it a ton of times library(tartan) #?call_scotty_demo call_scotty_demo("Big Bird")</code>

=== Step 7: And beyond! ===

In the above example we still didn’t cover a lot of things, including:

<ul>
<li> Including data in your package (that can be [http://r-pkgs.had.co.nz/data.html documented] too)
<code>r # suppose that I have a tarans dataset that I created - than I can save it this way usethis::use_data(tartans) # and document at usethis::use_r("data")</code> </li>
<li> Vignettes (long form documentation, knitr) </li>
<li> Compiled code (C, C++, Fortran…) </li>
<li> Creating a CRAN ready package </li>
<li> See [https://usethis.r-lib.org/reference/index.html <code>usethis</code>] to see all the package development steps that <code>usethis</code> automates. </li>
</ul>

File:Call scotty demo doc.png

2024-05-24T01:46:38Z

Skipper:

example of help file

File:Tartan description.png

2024-05-24T01:45:18Z

Skipper:

initialized description file

File:1200px-Tartan folder.png

2024-05-24T01:44:10Z

Skipper:

initial directory items

Parallel computing

2024-05-24T01:34:47Z

Skipper:

= When and why of parallel computing =

Parallel computing is a “hot” topic in computing and data science, but in many Statisticians’ daily lives it’s not too commonplace. This wiki article attempts to discuss some examples that might lead you to leveraging parallel computing as a statistician (often in the form of task-parallelization) and the trade-offs and potential power of parallel computing.

= General: Why parallel computing =

Some classic examples (expanded base on a list from [https://people.duke.edu/~ccc14/sta-663-2018/notebooks/S14A_Parallel_Programming_Introduction.html duke] for parallelization in statistics include

# simulations
## sensitivity testing
## parameter tuning
## Monte Carlo integration
# bootstrapping /other resampling
# Multiple MCMC chains

A lot of <code>R</code> or <code>python</code> code runs fast enough on a single processor. But at times, computations can be:

* '''cpu-bound''': Take too much cpu time
* '''memory-bound''': Take too much memory
* '''I/O-bound''': Take too much time to read/write from disk
* '''network-bound''': Take too much time to transfer

Parallelization directly helps with '''cpu-bound''' computations and can allow the user to leverage modern processor architectures that provide multiple cores on a single processor.

= General: When to parallelize (and when not to) =
Supposing you’re in a situation where your analysis’s computation is '''cpu-bound''', a secondary question is “can parallelization help speed this up?”. This question should capture
<ol>
<li>the amount of time you need to parallelize the process & combine the data,</li>
<li>how much increased overhead (in RAM & memory and cpu processing) is required to run and</li>
<li>if the process can even be parallelized.</li>
</ol>

== Potential to parallelize ==

<ol>
<li>'''Amdahl’s Law''': trade-oof between cores and parallelizable proportion of analysis In the computing community some of these trade-offs are captured by '''Amdahl’s Law''' which relates the increased speedup of your computation (y-axis) as a function of the number of cores (x-axis) and the proportion of computation that can be parallelized (see the colored lines). The image below visualizes this idea:

[[File:1200px-Parallelization.png|600px|frameless|thumb|center]]
 

For example, suppose you have to process a lot of early steps before running something that can be parallelized, and these early steps can’t be saved as they are too memory intensive (e.g. having more RAM than actually solid-state memory available), This might make the amount of computation that can be parallelized smalled (say 50%). In certain cases, this should discourage you to parallelize your process completely. This could lead your to do language-specific parallelization ([[parallelization_in_r|R parallel computing]] or [[parallelization_in_python|Python parallel computing]]) instead of higher-level parallelization (through [[Slurm]] or [[Bash_files|Bash files]] + [[Terminal_multiplexers|Terminal multiplexers]]).
</li>
<li>Memory Constraints

Outside considerations of Amdahl’s Law, the amount of RAM (especially on your personal computer), may give you pause. Often, when trying to alleviate cpu-bound constraints using parallelization one can run into memory-bound constraints. Specifically, running multiple processes intrinsically increases the amount of RAM memory used. this can actually make language-specific parallelization ([[parallelization_in_r|R parallel computing]] or [[parallelization_in_python|Python parallel computing]]) on personal computers '''unworkable''', and encourage utilizing resources with more RAM which can lead to using ([[Slurm]] or [[Bash files]] + [[Terminal_multiplexers]])-style parallelization.
</li></ol>

= Additional Reading =

As you can guess, there are more specific articles to help you get started/know some of what is available to actually do parallelization.

* Language-specific parallelization
** [[Parallelization_in_r|paralleliation with R]]
** [[Parallelization_in_python|parallelization with python]]
* Parallelization on Department servers
** You can use language-specific parallelization or combine [[Terminal multiplexers]] and [[Bash files]] to speed up the process
* [[Pittsburgh Super computer|PSC]]
** you’re best off learning the [[Slurm|Slurm workload manager]]

Parallel computing

2024-05-24T01:33:57Z

Skipper:

= When and why of parallel computing =

Parallel computing is a “hot” topic in computing and data science, but in many Statisticians’ daily lives it’s not too commonplace. This wiki article attempts to discuss some examples that might lead you to leveraging parallel computing as a statistician (often in the form of task-parallelization) and the trade-offs and potential power of parallel computing.

= General: Why parallel computing =

Some classic examples (expanded base on a list from [https://people.duke.edu/~ccc14/sta-663-2018/notebooks/S14A_Parallel_Programming_Introduction.html duke] for parallelization in statistics include

# simulations
## sensitivity testing
## parameter tuning
## Monte Carlo integration
# bootstrapping /other resampling
# Multiple MCMC chains

A lot of <code>R</code> or <code>python</code> code runs fast enough on a single processor. But at times, computations can be:

* '''cpu-bound''': Take too much cpu time
* '''memory-bound''': Take too much memory
* '''I/O-bound''': Take too much time to read/write from disk
* '''network-bound''': Take too much time to transfer

Parallelization directly helps with '''cpu-bound''' computations and can allow the user to leverage modern processor architectures that provide multiple cores on a single processor.

= General: When to parallelize (and when not to) =
[[File:1200px-Parallelization.png|600px|thumb|center]]
Supposing you’re in a situation where your analysis’s computation is '''cpu-bound''', a secondary question is “can parallelization help speed this up?”. This question should capture
<ol>
<li>the amount of time you need to parallelize the process & combine the data,</li>
<li>how much increased overhead (in RAM & memory and cpu processing) is required to run and</li>
<li>if the process can even be parallelized.</li>
</ol>

== Potential to parallelize ==

<ol>
<li>'''Amdahl’s Law''': trade-oof between cores and parallelizable proportion of analysis In the computing community some of these trade-offs are captured by '''Amdahl’s Law''' which relates the increased speedup of your computation (y-axis) as a function of the number of cores (x-axis) and the proportion of computation that can be parallelized (see the colored lines). The image below visualizes this idea:

[[File:Parallelization.png|frameless|600px|alt=amdahls law|amdahls law]]
 

For example, suppose you have to process a lot of early steps before running something that can be parallelized, and these early steps can’t be saved as they are too memory intensive (e.g. having more RAM than actually solid-state memory available), This might make the amount of computation that can be parallelized smalled (say 50%). In certain cases, this should discourage you to parallelize your process completely. This could lead your to do language-specific parallelization ([[parallelization_in_r|R parallel computing]] or [[parallelization_in_python|Python parallel computing]]) instead of higher-level parallelization (through [[Slurm]] or [[Bash_files|Bash files]] + [[Terminal_multiplexers|Terminal multiplexers]]).
</li>
<li>Memory Constraints

Outside considerations of Amdahl’s Law, the amount of RAM (especially on your personal computer), may give you pause. Often, when trying to alleviate cpu-bound constraints using parallelization one can run into memory-bound constraints. Specifically, running multiple processes intrinsically increases the amount of RAM memory used. this can actually make language-specific parallelization ([[parallelization_in_r|R parallel computing]] or [[parallelization_in_python|Python parallel computing]]) on personal computers '''unworkable''', and encourage utilizing resources with more RAM which can lead to using ([[Slurm]] or [[Bash files]] + [[Terminal_multiplexers]])-style parallelization.
</li></ol>

= Additional Reading =

As you can guess, there are more specific articles to help you get started/know some of what is available to actually do parallelization.

* Language-specific parallelization
** [[Parallelization_in_r|paralleliation with R]]
** [[Parallelization_in_python|parallelization with python]]
* Parallelization on Department servers
** You can use language-specific parallelization or combine [[Terminal multiplexers]] and [[Bash files]] to speed up the process
* [[Pittsburgh Super computer|PSC]]
** you’re best off learning the [[Slurm|Slurm workload manager]]

File:1200px-Parallelization.png

2024-05-24T01:33:10Z

Skipper:

amdahls law