Big Data Management
From Stat DS Computing Wiki
As your research experiments scale up in size, effective data management and storage becomes just as important as computational efficiency. This page will cover basic procedures and tools for moving, reading/writing, and storing large amounts of data.
Interacting with Big Data on Remote Servers
| Title | Description | Technical Level | Relevance |
|---|---|---|---|
| CLI for data management (TBW) | Introduces basic Linux commands for moving data to/from a remote server | ⭐⭐⚫⚫⚫ | All server workflows |
Tools
| Title | Description | Technical Level | Relevance |
|---|---|---|---|
| Globus | A graphical application for moving and sharing large files within and between insitutions | ⭐⚫⚫⚫⚫ | All server workflows |
