Big Data Management - Revision history

Ajshen at 02:04, 29 June 2026

2026-06-29T02:04:14Z

← Older revision		Revision as of 02:04, 29 June 2026
Line 1:		Line 1:
	[[Category:Beginner's_Guide_Scaling\|02]]		[[Category:Beginner's_Guide_Scaling\|03]]

	{{ScalingSidebar}}		{{ScalingSidebar}}

Ajshen: Undo revision 742 by Ajshen (talk)

2026-06-29T02:03:57Z

Undo revision 742 by Ajshen (talk)

← Older revision		Revision as of 02:03, 29 June 2026
Line 1:		Line 1:
	[[Category:Beginner'~~s_Guide_Reproducibility~~\|02]]		[[Category:Beginner's_Guide_Scaling\|02]]

	{{~~ReproducibilitySidebar~~}}		{{ScalingSidebar}}

	~~'''Dependencies''' are any external software, libraries, or packages that~~ your research ~~workflow requires~~ to ~~execute. For example~~, data ~~analysis in Python often relies on the NumPy~~ or ~~Pandas packages, while R workflows might rely heavily on~~ the ~~Tidyverse ecosystem~~.		As your research experiments scale up in size, effective data management and storage become just as important as computational efficiency. Working with large datasets—ranging from hundreds of gigabytes to several terabytes—requires specialized tools to move, store, and read/write data efficiently without bottlenecking your workflow or filling up the cluster.

	As your projects grow, it is critical to keep track of exactly which dependencies—and '''which specific versions''' of those dependencies—are being used. Unclear or insufficient dependency management is a leading cause of code breaking over time and is a major barrier to other researchers (or future you) reproducing your results.		This page covers basic procedures, best practices, and tools for handling large amounts of data on remote servers.

	This page covers basic procedures, best practices, and tools for ~~managing dependencies and encapsulating your computational environments~~.

	<div style="background-color: #fff3cd; border-left: 5px solid #ffc107; padding: 15px; margin-bottom: 20px; margin-top: 15px; overflow: hidden;">		<div style="background-color: #fff3cd; border-left: 5px solid #ffc107; padding: 15px; margin-bottom: 20px; margin-top: 15px; overflow: hidden;">
	'''Important: ~~Project Isolation~~'''<br>		'''Important: Storage Quotas and Scratch Space'''<br>
	~~A common beginner mistake is installing all packages globally into~~ your ~~computer's base environment~~. ~~You should strive~~ to ~~create a '''separate, isolated environment''' for every distinct research project. This prevents version conflicts~~ (~~e.g., Project A needs Python 3.9 but Project B requires Python 3.11~~) ~~and makes it much easier to document your work~~.		Unlike your personal laptop, department and shared servers usually have strict storage quotas in your personal `/home` directory. It is therefore important to budget available space accordingly and adhere to best practices (such as regular cleanup of temporary files) when using these servers.
	</div>		</div>

	== 1. ~~Virtual Environments~~ ==		== 1. Moving Data ==
	~~Virtual environments are lightweight, self-contained directories that contain a specific installation~~ of ~~a programming language~~ and ~~its packages~~. ~~They~~ are ~~the first line of defense for reproducible code on~~ your local machine.		Transferring gigabytes or terabytes of data over standard internet connections can be slow and prone to interruption. The tools below are designed to move data reliably between your local machine, the department servers, and external clusters.

	<div style="background-color: #f8f9fa; border-left: 4px solid #6c757d; padding: 15px; margin-bottom: 10px;">		<div style="background-color: #f8f9fa; border-left: 4px solid #6c757d; padding: 15px; margin-bottom: 10px;">
	'''[[~~Virtual Environment Introduction~~]]'''<br>		'''[[Command Line Data Transfer]]'''<br>
	Introduces ~~the core concepts of virtual environments for dependency management. Covers how to set up isolated workspaces using common~~ tools like <code>~~venv~~</code> or <code>~~conda~~</code> for ~~your analysis scripts~~.		Introduces basic Linux tools like <code>scp</code>, <code>rsync</code>, and <code>wget</code>. Ideal for moving small to medium-sized datasets or downloading data directly from the web to the server.
	</div>		</div>

	~~== 2. Containerization ==~~
	While virtual environments handle software packages, they do not capture the underlying operating system or system-level libraries. Containers solve this by packaging your code along with the ''entire'' system environment it needs to run, ensuring perfect reproducibility across different machines and servers.

	<div style="background-color: #f8f9fa; border-left: 4px solid #005A9C; padding: 15px; margin-bottom: 10px;">		<div style="background-color: #f8f9fa; border-left: 4px solid #005A9C; padding: 15px; margin-bottom: 10px;">
	'''[[~~Introduction to containers]] & [[Docker, and why-you-should-use-it-for-your-research\|Why Use Docker~~]]'''<br>		'''[[Globus]]'''<br>
	A ~~conceptual overview of what containers are, how they differ from traditional virtual machines~~, and ~~why Docker has become an essential tool~~ for ~~robust research workflows~~.		A graphical, web-based application for moving and sharing massive files within and between institutions. Globus handles network interruptions automatically and is highly recommended for terabyte-scale transfers.
	</div>		</div>

			== 2. File Formats and I/O ==
			When dealing with large data, traditional text formats like <code>.csv</code> take too long to read into memory and take up too much disk space. Transitioning to binary or columnar formats is a critical step in scaling up your research.

	<div style="background-color: #f8f9fa; border-left: 4px solid #75AADB; padding: 15px; margin-bottom: 10px;">		<div style="background-color: #f8f9fa; border-left: 4px solid #75AADB; padding: 15px; margin-bottom: 10px;">
	'''[[~~Tutorials\|Docker Basic Tutorial~~]]'''<br>		'''[[Efficient Data Formats]] (TBW)'''<br>
	~~A hands~~-~~on guide to building your first container~~. ~~Learn~~ how to write ~~a basic <code>Dockerfile</code>, build an image,~~ and ~~execute~~ your ~~code inside an isolated container environment~~.		An overview of high-performance file formats like Parquet, Feather, and HDF5. Explains how to read and write these formats in R and Python to drastically speed up your load times.
	</div>		</div>

	<div style="background-color: #f8f9fa; border-left: 4px solid #6f42c1; padding: 15px; margin-bottom: 10px;">		<div style="background-color: #f8f9fa; border-left: 4px solid #6f42c1; padding: 15px; margin-bottom: 10px;">
	'''[[~~Running Docker locally~~ and on ~~Hydras~~]]'''<br>		'''[[Data Compression on Linux]] (TBW)'''<br>
	~~Specific instructions for scaling up your containerized workflows~~. ~~Covers~~ how to ~~test your Docker containers locally~~ and ~~subsequently deploy them~~ on the ~~department's Hydra computing clusters~~.		How to use <code>tar</code>, <code>gzip</code>, and <code>zip</code> on the command line to compress large directories of files before transferring or archiving them.
			</div>

			== 3. Out-of-Memory Processing ==
			Sometimes, a dataset is simply too large to load into the RAM of even our largest servers. These tools allow you to query and manipulate datasets that exceed available memory.

			<div style="background-color: #f8f9fa; border-left: 4px solid #dc3545; padding: 15px; margin-bottom: 10px;">
			'''[[Out-of-Memory Data Manipulation]] (TBW)'''<br>
			An introduction to modern tools like DuckDB, Apache Arrow, or Dask. Learn how to filter, mutate, and aggregate massive datasets on disk before loading the smaller, processed results into RAM for statistical modeling.
	</div>		</div>

Ajshen: Undo revision 773 by Ajshen (talk)

2026-06-29T02:03:45Z

Undo revision 773 by Ajshen (talk)

← Older revision		Revision as of 02:03, 29 June 2026
Line 1:		Line 1:
	[[Category:Beginner'~~s_Guide_Scaling~~\|03]]		[[Category:Beginner's_Guide_Reproducibility\|02]]

	{{~~ScalingSidebar~~}}		{{ReproducibilitySidebar}}

	'''Dependencies''' are any external software, libraries, or packages that your research workflow requires to execute. For example, data analysis in Python often relies on the NumPy or Pandas packages, while R workflows might rely heavily on the Tidyverse ecosystem.		'''Dependencies''' are any external software, libraries, or packages that your research workflow requires to execute. For example, data analysis in Python often relies on the NumPy or Pandas packages, while R workflows might rely heavily on the Tidyverse ecosystem.

Ajshen at 02:00, 29 June 2026

2026-06-29T02:00:51Z

← Older revision		Revision as of 02:00, 29 June 2026
Line 1:		Line 1:
	[[Category:Beginner'~~s_Guide_Reproducibility~~\|02]]		[[Category:Beginner's_Guide_Scaling\|03]]

	{{~~ReproducibilitySidebar~~}}		{{ScalingSidebar}}

	'''Dependencies''' are any external software, libraries, or packages that your research workflow requires to execute. For example, data analysis in Python often relies on the NumPy or Pandas packages, while R workflows might rely heavily on the Tidyverse ecosystem.		'''Dependencies''' are any external software, libraries, or packages that your research workflow requires to execute. For example, data analysis in Python often relies on the NumPy or Pandas packages, while R workflows might rely heavily on the Tidyverse ecosystem.

Ajshen at 20:30, 28 June 2026

2026-06-28T20:30:56Z

← Older revision		Revision as of 20:30, 28 June 2026
Line 1:		Line 1:
	[[Category:Beginner'~~s_Guide_Scaling~~\|02]]		[[Category:Beginner's_Guide_Reproducibility\|02]]

	{{~~ScalingSidebar~~}}		{{ReproducibilitySidebar}}

	As your research ~~experiments scale up in size, effective data management and storage become just as important as computational efficiency~~. ~~Working with large datasets—ranging from hundreds of gigabytes to several terabytes—requires specialized tools to move, store~~, ~~and read/write~~ data ~~efficiently without bottlenecking your workflow~~ or ~~filling up~~ the ~~cluster~~.		'''Dependencies''' are any external software, libraries, or packages that your research workflow requires to execute. For example, data analysis in Python often relies on the NumPy or Pandas packages, while R workflows might rely heavily on the Tidyverse ecosystem.

	This page covers basic procedures, best practices, and tools for ~~handling large amounts of data on remote servers~~.		As your projects grow, it is critical to keep track of exactly which dependencies—and '''which specific versions''' of those dependencies—are being used. Unclear or insufficient dependency management is a leading cause of code breaking over time and is a major barrier to other researchers (or future you) reproducing your results.

			This page covers basic procedures, best practices, and tools for managing dependencies and encapsulating your computational environments.

	<div style="background-color: #fff3cd; border-left: 5px solid #ffc107; padding: 15px; margin-bottom: 20px; margin-top: 15px; overflow: hidden;">		<div style="background-color: #fff3cd; border-left: 5px solid #ffc107; padding: 15px; margin-bottom: 20px; margin-top: 15px; overflow: hidden;">
	'''Important: ~~Storage Quotas and Scratch Space~~'''<br>		'''Important: Project Isolation'''<br>
	~~Unlike~~ your ~~personal laptop~~, ~~department and shared servers usually have strict storage quotas in your personal `/home` directory~~. ~~It is therefore important to budget available space accordingly~~ and ~~adhere~~ to ~~best practices (such as regular cleanup of temporary files) when using these servers~~.		A common beginner mistake is installing all packages globally into your computer's base environment. You should strive to create a '''separate, isolated environment''' for every distinct research project. This prevents version conflicts (e.g., Project A needs Python 3.9 but Project B requires Python 3.11) and makes it much easier to document your work.
	</div>		</div>

	== 1. ~~Moving Data~~ ==		== 1. Virtual Environments ==
	~~Transferring gigabytes or terabytes~~ of ~~data over standard internet connections can be slow~~ and ~~prone to interruption~~. ~~The tools below~~ are ~~designed to move data reliably between~~ your local machine~~, the department servers, and external clusters~~.		Virtual environments are lightweight, self-contained directories that contain a specific installation of a programming language and its packages. They are the first line of defense for reproducible code on your local machine.

	<div style="background-color: #f8f9fa; border-left: 4px solid #6c757d; padding: 15px; margin-bottom: 10px;">		<div style="background-color: #f8f9fa; border-left: 4px solid #6c757d; padding: 15px; margin-bottom: 10px;">
	'''[[~~Command Line Data Transfer~~]]'''<br>		'''[[Virtual Environment Introduction]]'''<br>
	Introduces ~~basic Linux~~ tools like <code>~~scp~~</code>~~, <code>rsync</code>, and~~ <code>~~wget~~</code>~~. Ideal~~ for ~~moving small to medium-sized datasets or downloading data directly from the web to the server~~.		Introduces the core concepts of virtual environments for dependency management. Covers how to set up isolated workspaces using common tools like <code>venv</code> or <code>conda</code> for your analysis scripts.
	</div>		</div>

			== 2. Containerization ==
			While virtual environments handle software packages, they do not capture the underlying operating system or system-level libraries. Containers solve this by packaging your code along with the ''entire'' system environment it needs to run, ensuring perfect reproducibility across different machines and servers.

	<div style="background-color: #f8f9fa; border-left: 4px solid #005A9C; padding: 15px; margin-bottom: 10px;">		<div style="background-color: #f8f9fa; border-left: 4px solid #005A9C; padding: 15px; margin-bottom: 10px;">
	'''[[~~Globus~~]]'''<br>		'''[[Introduction to containers]] & [[Docker, and why-you-should-use-it-for-your-research\|Why Use Docker]]'''<br>
	A ~~graphical~~, ~~web-based application for moving and sharing massive files within and between institutions. Globus handles network interruptions automatically~~ and ~~is highly recommended~~ for ~~terabyte-scale transfers~~.		A conceptual overview of what containers are, how they differ from traditional virtual machines, and why Docker has become an essential tool for robust research workflows.
	</div>		</div>

	~~== 2. File Formats and I/O ==~~
	When dealing with large data, traditional text formats like <code>.csv</code> take too long to read into memory and take up too much disk space. Transitioning to binary or columnar formats is a critical step in scaling up your research.

	<div style="background-color: #f8f9fa; border-left: 4px solid #75AADB; padding: 15px; margin-bottom: 10px;">		<div style="background-color: #f8f9fa; border-left: 4px solid #75AADB; padding: 15px; margin-bottom: 10px;">
	'''[[~~Efficient Data Formats~~]] ~~(TBW)~~'''<br>		'''[[Tutorials\|Docker Basic Tutorial]]'''<br>
	~~An overview of high~~-~~performance file formats like Parquet, Feather, and HDF5~~. ~~Explains~~ how to ~~read and~~ write ~~these formats in R~~ and ~~Python to drastically speed up~~ your ~~load times~~.		A hands-on guide to building your first container. Learn how to write a basic <code>Dockerfile</code>, build an image, and execute your code inside an isolated container environment.
	</div>		</div>

	<div style="background-color: #f8f9fa; border-left: 4px solid #6f42c1; padding: 15px; margin-bottom: 10px;">		<div style="background-color: #f8f9fa; border-left: 4px solid #6f42c1; padding: 15px; margin-bottom: 10px;">
	'''[[~~Data Compression on Linux]] (TBW)'''<br>~~		'''[[Running Docker locally and on Hydras]]'''<br>
	~~How to use <code>tar</code>, <code>gzip</code>,~~ and ~~<code>zip</code>~~ on ~~the command line to compress large directories of files before transferring or archiving them.~~		Specific instructions for scaling up your containerized workflows. Covers how to test your Docker containers locally and subsequently deploy them on the department's Hydra computing clusters.
	~~</div>~~

	~~== 3. Out-of-Memory Processing ==~~
	~~Sometimes, a dataset is simply too large to load into the RAM of even our largest servers. These tools allow you to query and manipulate datasets that exceed available memory.~~

	~~<div style="background-color: #f8f9fa; border-left: 4px solid #dc3545; padding: 15px; margin-bottom: 10px;">~~
	~~'''[[Out-of-Memory Data Manipulation~~]] ~~(TBW)~~'''<br>
	~~An introduction to modern tools like DuckDB, Apache Arrow, or Dask~~. ~~Learn~~ how to ~~filter, mutate,~~ and ~~aggregate massive datasets~~ on ~~disk before loading~~ the ~~smaller, processed results into RAM for statistical modeling~~.
	</div>		</div>

Ajshen: /* 2. File Formats and I/O */

2026-06-27T06:20:22Z

2. File Formats and I/O

← Older revision		Revision as of 06:20, 27 June 2026
Line 29:		Line 29:

	<div style="background-color: #f8f9fa; border-left: 4px solid #75AADB; padding: 15px; margin-bottom: 10px;">		<div style="background-color: #f8f9fa; border-left: 4px solid #75AADB; padding: 15px; margin-bottom: 10px;">
	'''[[Efficient Data Formats (TBW)]]'''<br>		'''[[Efficient Data Formats]] (TBW)'''<br>
	An overview of high-performance file formats like Parquet, Feather, and HDF5. Explains how to read and write these formats in R and Python to drastically speed up your load times.		An overview of high-performance file formats like Parquet, Feather, and HDF5. Explains how to read and write these formats in R and Python to drastically speed up your load times.
	</div>		</div>

	<div style="background-color: #f8f9fa; border-left: 4px solid #6f42c1; padding: 15px; margin-bottom: 10px;">		<div style="background-color: #f8f9fa; border-left: 4px solid #6f42c1; padding: 15px; margin-bottom: 10px;">
	'''[[Data Compression on Linux (TBW)]]'''<br>		'''[[Data Compression on Linux]] (TBW)'''<br>
	How to use <code>tar</code>, <code>gzip</code>, and <code>zip</code> on the command line to compress large directories of files before transferring or archiving them.		How to use <code>tar</code>, <code>gzip</code>, and <code>zip</code> on the command line to compress large directories of files before transferring or archiving them.
	</div>		</div>

Ajshen: /* 2. File Formats and I/O */

2026-06-27T06:20:06Z

2. File Formats and I/O

← Older revision		Revision as of 06:20, 27 June 2026
Line 29:		Line 29:

	<div style="background-color: #f8f9fa; border-left: 4px solid #75AADB; padding: 15px; margin-bottom: 10px;">		<div style="background-color: #f8f9fa; border-left: 4px solid #75AADB; padding: 15px; margin-bottom: 10px;">
	'''[[Efficient Data Formats]]'''<br>		'''[[Efficient Data Formats (TBW)]]'''<br>
	An overview of high-performance file formats like Parquet, Feather, and HDF5. Explains how to read and write these formats in R and Python to drastically speed up your load times.		An overview of high-performance file formats like Parquet, Feather, and HDF5. Explains how to read and write these formats in R and Python to drastically speed up your load times.
	</div>		</div>

	<div style="background-color: #f8f9fa; border-left: 4px solid #6f42c1; padding: 15px; margin-bottom: 10px;">		<div style="background-color: #f8f9fa; border-left: 4px solid #6f42c1; padding: 15px; margin-bottom: 10px;">
	'''[[Data Compression on Linux]]'''<br>		'''[[Data Compression on Linux (TBW)]]'''<br>
	How to use <code>tar</code>, <code>gzip</code>, and <code>zip</code> on the command line to compress large directories of files before transferring or archiving them.		How to use <code>tar</code>, <code>gzip</code>, and <code>zip</code> on the command line to compress large directories of files before transferring or archiving them.
	</div>		</div>

Ajshen: /* 1. Moving Data */

2026-06-27T06:19:22Z

1. Moving Data

← Older revision		Revision as of 06:19, 27 June 2026
Line 23:		Line 23:
	'''[[Globus]]'''<br>		'''[[Globus]]'''<br>
	A graphical, web-based application for moving and sharing massive files within and between institutions. Globus handles network interruptions automatically and is highly recommended for terabyte-scale transfers.		A graphical, web-based application for moving and sharing massive files within and between institutions. Globus handles network interruptions automatically and is highly recommended for terabyte-scale transfers.
	~~</div>~~

	~~<div style="background-color: #f8f9fa; border-left: 4px solid #28a745; padding: 15px; margin-bottom: 10px;">~~
	~~'''[[Rclone]] (TBW)'''<br>~~
	~~A command-line tool that acts like <code>rsync</code> for cloud storage. Essential if your dataset is currently hosted on Google Drive, AWS S3, or Dropbox and needs to be pulled down to the cluster.~~
	</div>		</div>

Ajshen at 05:58, 27 June 2026

2026-06-27T05:58:33Z

← Older revision		Revision as of 05:58, 27 June 2026
Line 9:		Line 9:
	<div style="background-color: #fff3cd; border-left: 5px solid #ffc107; padding: 15px; margin-bottom: 20px; margin-top: 15px; overflow: hidden;">		<div style="background-color: #fff3cd; border-left: 5px solid #ffc107; padding: 15px; margin-bottom: 20px; margin-top: 15px; overflow: hidden;">
	'''Important: Storage Quotas and Scratch Space'''<br>		'''Important: Storage Quotas and Scratch Space'''<br>
	Unlike your personal laptop, department and shared servers usually have strict storage quotas in your personal `/home` directory. It is therefore important to budget available space accordingly and adhere to best practices (such as ~~regularly~~ cleanup of temporary files).		Unlike your personal laptop, department and shared servers usually have strict storage quotas in your personal `/home` directory. It is therefore important to budget available space accordingly and adhere to best practices (such as regular cleanup of temporary files) when using these servers.
	</div>		</div>

Ajshen at 05:58, 27 June 2026

2026-06-27T05:58:09Z

← Older revision		Revision as of 05:58, 27 June 2026
Line 1:		Line 1:
	As your research experiments scale up in size, effective data management and storage becomes just as important as computational efficiency. This page will cover basic procedures and tools for moving, reading/writing, and storing large amounts of data.		[[Category:Beginner's_Guide_Scaling\|02]]

	~~Back to [[Beginner's Guide to Computation in Statistical Research]]~~		{{ScalingSidebar}}

	~~== Interacting with Big Data on Remote Servers ==~~		As your research experiments scale up in size, effective data management and storage become just as important as computational efficiency. Working with large datasets—ranging from hundreds of gigabytes to several terabytes—requires specialized tools to move, store, and read/write data efficiently without bottlenecking your workflow or filling up the cluster.
	~~{\| class="wikitable"~~
	\|+
	~~!Title~~
	~~!Description~~
	~~!Technical Level~~
	~~!Relevance~~
	\|-
	~~\|CLI for~~ data management ~~(TBW)~~
	~~\|Introduces basic Linux commands for moving data~~ to/~~from a remote server~~
	~~\|⭐⭐⚫⚫⚫~~
	~~\|All server workflows~~
	\|-
	\|
	\|
	\|
	\|
	\|}

	== ~~Tools~~ ==		This page covers basic procedures, best practices, and tools for handling large amounts of data on remote servers.
	~~{\| class~~="~~wikitable~~"
	~~!Title~~		<div style="background-color: #fff3cd; border-left: 5px solid #ffc107; padding: 15px; margin-bottom: 20px; margin-top: 15px; overflow: hidden;">
	~~!Description~~		'''Important: Storage Quotas and Scratch Space'''<br>
	~~!Technical Level~~		Unlike your personal laptop, department and shared servers usually have strict storage quotas in your personal `/home` directory. It is therefore important to budget available space accordingly and adhere to best practices (such as regularly cleanup of temporary files).
	~~!Relevance~~		</div>
	\|-
	\|[[Globus]]		== 1. Moving Data ==
	\|A graphical application for moving and sharing ~~large~~ files within and between ~~insitutions~~		Transferring gigabytes or terabytes of data over standard internet connections can be slow and prone to interruption. The tools below are designed to move data reliably between your local machine, the department servers, and external clusters.
	~~\|⭐⚫⚫⚫⚫~~
	~~\|All server workflows~~		<div style="background-color: #f8f9fa; border-left: 4px solid #6c757d; padding: 15px; margin-bottom: 10px;">
	\|-		'''[[Command Line Data Transfer]]'''<br>
	\|		Introduces basic Linux tools like <code>scp</code>, <code>rsync</code>, and <code>wget</code>. Ideal for moving small to medium-sized datasets or downloading data directly from the web to the server.
	\|		</div>
	\|
	\|		<div style="background-color: #f8f9fa; border-left: 4px solid #005A9C; padding: 15px; margin-bottom: 10px;">
	\|}		'''[[Globus]]'''<br>
			A graphical, web-based application for moving and sharing massive files within and between institutions. Globus handles network interruptions automatically and is highly recommended for terabyte-scale transfers.
			</div>

			<div style="background-color: #f8f9fa; border-left: 4px solid #28a745; padding: 15px; margin-bottom: 10px;">
			'''[[Rclone]] (TBW)'''<br>
			A command-line tool that acts like <code>rsync</code> for cloud storage. Essential if your dataset is currently hosted on Google Drive, AWS S3, or Dropbox and needs to be pulled down to the cluster.
			</div>

			== 2. File Formats and I/O ==
			When dealing with large data, traditional text formats like <code>.csv</code> take too long to read into memory and take up too much disk space. Transitioning to binary or columnar formats is a critical step in scaling up your research.

			<div style="background-color: #f8f9fa; border-left: 4px solid #75AADB; padding: 15px; margin-bottom: 10px;">
			'''[[Efficient Data Formats]]'''<br>
			An overview of high-performance file formats like Parquet, Feather, and HDF5. Explains how to read and write these formats in R and Python to drastically speed up your load times.
			</div>

			<div style="background-color: #f8f9fa; border-left: 4px solid #6f42c1; padding: 15px; margin-bottom: 10px;">
			'''[[Data Compression on Linux]]'''<br>
			How to use <code>tar</code>, <code>gzip</code>, and <code>zip</code> on the command line to compress large directories of files before transferring or archiving them.
			</div>

			== 3. Out-of-Memory Processing ==
			Sometimes, a dataset is simply too large to load into the RAM of even our largest servers. These tools allow you to query and manipulate datasets that exceed available memory.

			<div style="background-color: #f8f9fa; border-left: 4px solid #dc3545; padding: 15px; margin-bottom: 10px;">
			'''[[Out-of-Memory Data Manipulation]] (TBW)'''<br>
			An introduction to modern tools like DuckDB, Apache Arrow, or Dask. Learn how to filter, mutate, and aggregate massive datasets on disk before loading the smaller, processed results into RAM for statistical modeling.
			</div>