Abhigyan Chakraborty

Project Organization and Managing Scientific Data

Phase 2: The Research Infrastructure — Part 3

(Part 7 of series Blueprint for a Modern Research Computing Environment)

Follow me :

LinkedIn    Website    Website

Quick Summary

This article establishes a reusable project structure for research code, sets the rule that raw data is never modified, and separates data storage from code storage entirely. It also covers naming conventions, writing a README, what belongs on GitHub versus what doesn’t, and where research data can live — locally, in the cloud, or published with a citable DOI.

Objective

You’ve been building project_1 throughout this series. So far, the focus has been on tools — installing, configuring, connecting. But as your projects grow, how you organize them matters as much as the code itself.

A well-organized project is easier to understand, easier to share, and easier to come back to after six months away.

By the end, you’ll have:

Content

💡 Getting Unstuck (Expand for AI Troubleshooting Prompts)

AI tools (like Claude, Grok, Gemini, or ChatGPT) give the best troubleshooting advice when they have the exact text of the tutorial. To ensure the AI understands what you are trying to build, we will give it the actual file.

1. Download the tutorial file

2. Upload it to your AI

3. Ask for Help Copy and paste this exact prompt into the chat along with your file:

I have attached the markdown file for the tutorial I am following. Please read it so you understand the specific environment I am trying to build.

I need help with the following:

Step [X]: [paste exact step text from the blog]

Command I ran: [paste exact command]

What happened: [paste full output/error — if the terminal was truly blank, say so explicitly]

Please help me troubleshoot and fix this error. You can use your general knowledge to solve the problem, but your solution MUST align with the architecture and tools taught in the attached file. Do not suggest alternative setups that contradict the tutorial. Once fixed, tell me what to do next in the article.

To go deeper on a step before you run a command:

I have attached the Part7.md file. Look at Step [X] and explain exactly what the command does and why we are doing it before I run it.

Think of this series as the roadmap and your AI assistant as your learning companion.

Prerequisites

Step 1 — Why Project Organization Matters

Imagine coming back to a project after six months. Files named script_final_v3_FINAL.py. Data mixed in with code. No README. No idea which script to run first.

This is more common than it should be — especially in research, where the pressure is to get results, not to organize files. Good organization costs almost nothing upfront and saves enormous time later.

Step 2 — A Standard Project Structure

Here is a structure that works for most research projects:

project_1/
├── notebooks/        ← .ipynb files for exploration
├── scripts/          ← .py files for automation and pipelines
├── results/
│   ├── figures/      ← plots and visualizations
│   └── outputs/      ← model outputs, tables, summaries
├── environment.yml   ← Conda environment definition
├── .gitignore        ← files Git should ignore
└── README.md         ← what this project is and how to use it

The key principle: separate your inputs (data), your process (notebooks and scripts), and your outputs (results).

Create this structure:

cd ~/project_1
mkdir -p notebooks scripts results/figures results/outputs
touch README.md

Step 3 — The Raw Data Rule

Never modify raw data. Ever.

Your raw data folder is read-only in practice — you load from it, you never write to it. If you need to clean or transform data, save the result to a separate processed folder.

Step 4 — Where Data Lives

We follow this split:

Keep data outside project_1 because it is often large, doesn’t need version control, and is shared across projects.

Create this on your Windows drive (or Linux equivalent):

mkdir -p /mnt/d/datasets/project_1/raw
mkdir -p /mnt/d/datasets/project_1/processed

Step 5 — Naming Conventions

Step 6 — README.md

A README is the front door of your project. Open and fill it in:

cd ~/project_1
nano README.md

Include sections for: What this is, Data location, How to run, Environment setup, and Results.

Step 7 — What Goes on GitHub and What Doesn’t

GitHub is for code — not data.

Add large items to .gitignore:

nano ~/project_1/.gitignore

Step 8 — Storage Recommendations

Step 9 — Domain-Specific File Formats

If your research involves scientific data, you’ll likely encounter formats like netCDF, HDF5, and GRIB. These require specific libraries like xarray or h5py. For geoscientific workflows, explore Project Pythia.

What’s Next

What You’ve Done:

Next: Part 8 — Enabling GPU Computing in WSL2 and Linux with CUDA | Previous: Part 6 — Git and GitHub for Reproducible Research

All Blogs

This site is open source. Improve this page.