Main Page: Difference between revisions

From Stat DS Computing Wiki
← Older edit
VisualWikitext
Jacobg2 (talk | contribs)
Administrators, administrator, Bureaucrats, Interface administrators, students, Administrators, Users
336 edits
 
(25 intermediate revisions by 3 users not shown)
Line 1: Line 1:
Anyone can read this wiki.  If you want to write/edit stuff in the wiki, you will need to log in with your account provided by the Statistics & Data Science Department computing group. This Wiki is intended to be a repository for instruction, information and ideas related to the many computing environments available to members and affiliates of the Statistics & Data Science Department.
<div style="background-color: #f8f9fa; border: 5px solid #17a2b8; padding: 30px; margin-bottom: 25px; text-align: center;">
<span style="font-size: 150%; font-weight: bold;">Looking for a specific answer to a quick question? Consider using our [https://util.stat.cmu.edu/statchat/ automated support chat].</span>
</div>


== Getting to know the computing resources & getting support ==
Welcome to the CMU Department of Statistics and Data Science Computing Wiki! To get the most out of this resource, please note the following:


If you're interested in knowing about what resources you have access to (and/or) what resources you have access to that might solve a problem you're having, please visit our [[What_resources_when|"what resources when" article]].
<div style="background-color: #e2e3e5; border: 5px solid #6c757d; padding: 15px; margin-bottom: 20px; margin-top: 15px;">
'''Purpose & Contribution'''<br>
The main purpose of this wiki is to familiarize readers with the computing resources provided by the department.
* '''Newcomers:''' Please skim the [[Beginner's Guide to Computation in Statistical Research]] to familiarize yourself with statistical computing tools and best practices as you begin you research.
* '''Computational Researchers:''' See the [[Article Catalog]] for a full selection of topics and pages and the [[Directory of Computing Resources for Statistical Research]] for available computational resources. You can also use the search bar at the top of this page. For one-off questions, consider using the [https://util.stat.cmu.edu/statchat/ Support Chat]
* '''Contributors and Maintainers:''' Feel free to leave a comment on any page, or create your own pages. Please see [[Generic Wiki Information]] for more information.
</div>


To get support that is beyond this website please attend office hours below and/or email [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu] for support (for more details about how the support ticketing works through [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu] + ServiceNow please see the [[Service_now|ServiceNow]] article).
== Important Announcements ==
Before using any department computing resources, please see [[Research Computing Dos and Don'ts]]<div style="background-color: #fff8e6; border-left: 4px solid #ffc107; padding: 15px; margin-bottom: 20px;">
* '''[2026-09-11] New Article:''' [[End-to-end AI Scientist Trials (Methods/Applied Research)]]
* '''[2026-08-19] New Article:''' Automating experiments using [[Agentic Workflow via GitHub Issues]]
* '''[2026-04-10] New Article:''' Introducing [[Globus]], a tool for sharing large files within and between institutions.
* '''[2026-04-01] PSA:''' If you use VS Code, look for "[https://marketplace.visualstudio.com/items?itemName=ms-python.vscode-python-envs Python Environments]" in your extensions and '''uninstall it immediately''' due to environment conflicts.
* '''[2026-04-01] Updated Article:''' The [[Linux and department servers]] page has been updated with instructions on how to use <code>/scratch</code> to avoid disk I/O slowdowns. See [[Linux and department servers#Using /scratch]].
</div>


Reset your StatDS password here - https://util.stat.cmu.edu/passwordreset/
== Quick Start ==
<div style="display: grid; grid-template-columns: repeat(auto-fit, minmax(500px, 1fr))">
<div style="background-color: #f8f9fa; border-left: 4px solid #005A9C; padding: 15px; margin-bottom: 10px;">
'''[[Beginner's Guide to Computation in Statistical Research]]'''<br>
The essential starting point for understanding how to scale your statistical workflows and choose the right tools for your research.
</div>


== Department Contacts ==
<div style="background-color: #f8f9fa; border-left: 4px solid #28a745; padding: 15px; margin-bottom: 10px;">
'''[[Directory of Computing Resources for Statistical Research]]'''<br>
A comprehensive index of all available clusters, servers, and computing environments accessible to department members.
</div>


• [mailto:events@stat.cmu.edu events@stat.cmu.edu]: seminars, conferences & workshops, and calendar updates
<div style="background-color: #f8f9fa; border-left: 4px solid #6f42c1; padding: 15px; margin-bottom: 10px;">
'''[[Department Tools and Services]]'''<br>
Additional department tools, printing, web hosting, software licenses, and specialized computing services.
</div>


• [mailto:news@stat.cmu.edu news@stat.cmu.edu]: dept/dept member-related public news & announcements
<div style="background-color: #f8f9fa; border-left: 4px solid #eff07d; padding: 15px; margin-bottom: 10px;">
'''[[Article Catalog]]'''<br>
A comprehensive list of articles organized by topic
</div>


• [mailto:lists@stat.cmu.edu lists@stat.cmu.edu]: department email lists
</div>


• [mailto:webmaster@stat.cmu.edu webmaster@stat.cmu.edu]: website updates
== Support Resources ==
For assistance with issues not covered in this wiki, please reach out to the following contacts:


• [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu]: computer/tech requests
<div style="display: grid; grid-template-columns: repeat(auto-fit, minmax(500px, 1fr))">
<div style="background-color: #f8f9fa; border-left: 4px solid #6c757d; padding: 15px; margin-bottom: 10px;">
'''General Tech Support'''<br>
For general issues with department computing resources:<br>
'''Email:''' [mailto:remarks@stat.cmu.edu remarks@stat.cmu.edu]
</div>


• [mailto:rooms@stat.cmu.edu rooms@stat.cmu.edu]: request space for meetings and TA office hours
<div style="background-color: #f8f9fa; border-left: 4px solid #17a2b8; padding: 15px; margin-bottom: 10px;">
'''Research Project Assistance (Graduate Students Only)'''<br>
Live assistance from Alex Shen, PhD Computing TA, for project-specific computing questions:<br>
'''Office Hours:''' Mondays 3:00 PM - 4:00 PM @ Baker Hall 154A<br>
'''Contact:''' [mailto:ajshen@andrew.cmu.edu ajshen@andrew.cmu.edu] or reach out directly via Slack.
</div>


• [mailto:reimbursements@stat.cmu.edu reimbursements@stat.cmu.edu]: travel and reimbursement requests
<div style="background-color: #f8f9fa; border-left: 4px solid #17a2b8; padding: 15px; margin-bottom: 10px;">
'''Onboarding & Troubleshooting (Graduate Students Only)'''<br>
Live assistance for setting up and troubleshooting computing resources:<br>
'''Carl Skipper:''' Office hours by appointment. Request via [mailto:skipper@andrew.cmu.edu skipper@andrew.cmu.edu]<br>
'''Jake Gordon:''' Office hours by appointment. Request via [mailto:jacobg2@andrew.cmu.edu jacobg2@andrew.cmu.edu]
</div>
</div>


== Computing Office Hours ==
== Suggestions Box and Todos ==


These office hour opportunities provide a dedicated time for you to come ask questions about research computing.&nbsp; These include code advice, how to access certain resources, what the best resources are for your project, or to learn about new connections to help you progress in your education and research. Staff Office hour times will be sent out each week on Monday as a calendar invite; accept any of the invitations to add it to your personal calendar. PhD TA Office Hours are below, unless a different time is&nbsp;communicated.
<div style="background-color: #aff0ff; border: 4px solid #17a2b8; padding: 15px; margin-bottom: 10px;">
* '''[New Article]''' Beginner's Guide -> Scaling -> Version Control -> Experiment Management
* '''[New Article]''' Beginner's Guide -> Scaling -> Big Data Management -> Articles about file formats and compression
* '''[New Article]''' Beginner's Guide -> Software Development -> Package Creation -> Writing Documentation
* '''[New Article]''' Beginner's Guide -> Software Development -> Package Repositories


#PhD TA Office Hours for the first half of summer 2025 are held by Elizabeth Cucuzzella in FMS Room 320 on Mondays from 2:30 pm to 4:30 pm. PhD students can also message on the #computing channel on slack.
</div>
#Staff Office Hours are held by appointment by Carl Skipper in BH 129G.  Schedule via [mailto:lists@stat.cmu.edu email].


See [[Computing_office_hour_usage|computing office hour usage]] for what office hours can be used for.
This section contains work in progress and future suggestions for the wiki. If you would like to leave a suggestions, you may:
 
* Directly contact the Computing TA
== Computing Wiki Pages ==
* Leave a comment on a specific wiki article with your suggestions
 
* Leave a comment on this page with your suggestions
=== Managing servers and environments ===
<br>
 
==== Introductory Tutorials ====
[[Basic Linux Commands|Introduction to Basic Linux Commands]]
 
==== Virtual environments ====
*[[Virtual Environment Introduction|Why use virtual environments?]]
*[https://info.stat.cmu.edu/index.php?title=Mamba Mamba (Conda) environments].
*R's [[Renv]]
*Python's [[Python_virtual_environments|virtual environments (2 options)]]
*Old instructions for [[Conda with Python (Reference Only)|conda with Python]]
 
==== Containers ====
*[[Introduction to containers]]
*[[Docker, and why-you-should-use-it-for-your-research]]
*[[Tutorials|Basic tutorial]]
*[[Running Docker locally and on Hydras]]
 
=== Department computing resources ===
 
==== Printing ====
*[[Department_Printers|Department Printers]]
 
==== Web Hosting ====
#[[How to host your webpage under stat.cmu.edu]]
#[[Bulding your website with Hugo]]
 
==== Servers ====
*[[Department Servers]]. 
*#[[Linux and department servers|Using the Servers Overview]] -This wiki introduces you to the '''hydra''' servers, basic Linux commands, and more.
*#[[Terminal_multiplexers|Terminal multiplexers (screen and tmux)]]. This wiki presents tools for use on the ''hydra'' that can allow you to run jobs even when your not on the server, and much much more!
*#[[Python on the server]]. This wiki introduces uses command line approaches to running <code>python</code> on a virtual machine, and how to use jupyter notebooks through the server as well.
*#[[R on the server]]. This wiki introduces uses command line approaches to running <code>R</code> on a virtual machine
*#[https://info.stat.cmu.edu/index.php?title=Department_Servers#Software Software] - overview of the software installed and ready for use.
*#[https://info.stat.cmu.edu/index.php?title=Monitoring_with_Grafana Monitor Server Usage]
 
*[[Department GPU Servers]], for small jobs needing 1 GPU.
*[[GPU Cluster]], for larger jobs that may need multiple GPUs and many CPUs.
*[[PSC]], the Pittsburgh Supercomputing Center. Free accounts and usage credits are available
*#[[ACCESS]]
*#[[Slurm]]. This wiki introduces the job management system uses at PSC.
*#[[Parallelization_on_PSC/Bridges-2]]
*#[[Access PSC compute nodes directly from VSCode]]
*[[Open OnDemand]]
 
=== Development guides ===
 
====[[Parallel_computing|Parallelization]]====
*[[Parallelization_in_r|R-specific parallelization]]
*[[Parallelization_in_python|Python-specific parallelization]]
 
==== Package development ====
*[[Developing_R_packages|R package development]]
*[[Developing_python_packages|Python package development]]
[[Terminal automation]]. This wiki introduces approaches to automate Linux/OS command line calls.
*[[Bash files]]. This wiki presents potential uses of Bash files including automated file/folder creation, parallelization and much more!
*[[Make]]. This wiki presents potential uses of <code>make</code> and Makefiles including automated workflow and dependency tracking
*[[Jumping to frequently used folders and automating environment setups]]
 
==== Version Control ====
See the guide provided in the site of [https://36-750.github.io/tools/version-control/ 36750 - Statistical Computing]
 
==== Reference management ====
#Zotero
#Mendeley
 
==== Integrated Development Environments (IDEs) ====
#[[VSCode|VSCode]]
#PyCharm
#RStudio
#Eclipse
 
==== Text Editors ====
#Vim
#Nano
#Emacs
#Atom
#Sublime
 
See also the guide provided in the site of [https://36-750.github.io/tools/editors/ 36750 - Statistical Computing]
 
==== AI Tools ====
* [[ChatGPT and other LLMs]]
 
*[[github_copilot|GitHub Copilot]]
 
==== Miscellaneous====
#[[Windows_Subsystem_for_Linux_(WSL)_and_Python|Windows&nbsp;Subsystem for Linux (WSL) and Python]]
#[[Amphetamine|Amphetamine: How to keep your computer awake]]
 
===General Department Computing Resources===
 
*Fall 2021 [[Computing_seminar_resources|Computing Seminar Resources]] (video and slides)
 
==Ideas for the Wiki==
 
Please see the [[TODO|TODO page]] to post suggestions and errata.
 
==Using the Wiki==
 
*Here is some&nbsp;[[Generic_Wiki_Information|Generic Wiki Information]]&nbsp;that you may be interested in, such as a users' guide and other resources for Wiki users.
 
== Archived Pages ==
These area pages that are potentially useful for some users, but no longer reflect the computing resources of the department writ large.
 
Azure
#[[Azure_getting_started|Azure: getting started]]. This wiki walks you through the steps to create a virtual machine through Azure
#[[How to obtain a virtual machine with a GPU]]
#[[Attach a disk to a virtual machine]]

Latest revision as of 03:41, 19 September 2026

Looking for a specific answer to a quick question? Consider using our automated support chat.

Welcome to the CMU Department of Statistics and Data Science Computing Wiki! To get the most out of this resource, please note the following:

Purpose & Contribution
The main purpose of this wiki is to familiarize readers with the computing resources provided by the department.

Important Announcements

Before using any department computing resources, please see Research Computing Dos and Don'ts

Quick Start

Beginner's Guide to Computation in Statistical Research
The essential starting point for understanding how to scale your statistical workflows and choose the right tools for your research.

Directory of Computing Resources for Statistical Research
A comprehensive index of all available clusters, servers, and computing environments accessible to department members.

Department Tools and Services
Additional department tools, printing, web hosting, software licenses, and specialized computing services.

Article Catalog
A comprehensive list of articles organized by topic

Support Resources

For assistance with issues not covered in this wiki, please reach out to the following contacts:

General Tech Support
For general issues with department computing resources:
Email: remarks@stat.cmu.edu

Research Project Assistance (Graduate Students Only)
Live assistance from Alex Shen, PhD Computing TA, for project-specific computing questions:
Office Hours: Mondays 3:00 PM - 4:00 PM @ Baker Hall 154A
Contact: ajshen@andrew.cmu.edu or reach out directly via Slack.

Onboarding & Troubleshooting (Graduate Students Only)
Live assistance for setting up and troubleshooting computing resources:
Carl Skipper: Office hours by appointment. Request via skipper@andrew.cmu.edu
Jake Gordon: Office hours by appointment. Request via jacobg2@andrew.cmu.edu

Suggestions Box and Todos

  • [New Article] Beginner's Guide -> Scaling -> Version Control -> Experiment Management
  • [New Article] Beginner's Guide -> Scaling -> Big Data Management -> Articles about file formats and compression
  • [New Article] Beginner's Guide -> Software Development -> Package Creation -> Writing Documentation
  • [New Article] Beginner's Guide -> Software Development -> Package Repositories

This section contains work in progress and future suggestions for the wiki. If you would like to leave a suggestions, you may:

  • Directly contact the Computing TA
  • Leave a comment on a specific wiki article with your suggestions
  • Leave a comment on this page with your suggestions


Retrieved from "https://info.stat.cmu.edu/index.php?title=Main_Page&oldid=977"