Abhigyan Chakraborty

Linux Essentials for HPC and Remote Servers

Phase 2: The Research Infrastructure — Part 1

(Part 5 of series Blueprint for a Modern Research Computing Environment)

Follow me :

LinkedIn    Website    Website

Quick Summary

Unlike previous parts, this article serves as a map rather than a build tutorial. It covers directory organization, file management, permissions, installing system software with apt, environment variables, SSH, and the basics of HPC clusters and SLURM job submission — the concepts researchers need once they move beyond their local machine to remote servers.

Objective

In previous parts, you built a concrete environment on your local machine. Here, the goal is different: this is a map. As a researcher, you’ll eventually work on remote servers and university HPC (High Performance Computing) clusters. This article provides the foundational knowledge needed to navigate these environments confidently.

By the end, you’ll have:

Content

💡 Getting Unstuck (Expand for AI Troubleshooting Prompts)

AI tools (like Claude, Grok, Gemini, or ChatGPT) give the best troubleshooting advice when they have the exact text of the tutorial. To ensure the AI understands what you are trying to build, we will give it the actual file.

1. Download the tutorial file

2. Upload it to your AI

3. Ask for Help Copy and paste this exact prompt into the chat along with your file:

I have attached the markdown file for the tutorial I am following. Please read it so you understand the specific environment I am trying to build.

I need help with the following:

Step [X]: [paste exact step text from the blog]

Command I ran: [paste exact command]

What happened: [paste full output/error — if the terminal was truly blank, say so explicitly]

Please help me troubleshoot and fix this error. You can use your general knowledge to solve the problem, but your solution MUST align with the architecture and tools taught in the attached file. Do not suggest alternative setups that contradict the tutorial. Once fixed, tell me what to do next in the article.

To go deeper on a step before you run a command:

I have attached the Part5.md file. Look at Step [X] and explain exactly what the command does and why we are doing it before I run it.

Think of this series as the roadmap and your AI assistant as your learning companion.

Prerequisites

Section 1 — Creating and Organizing Directories

Create nested directories in one command:

mkdir -p ~/project_1/data/raw
mkdir -p ~/project_1/data/processed
mkdir -p ~/project_1/scripts
mkdir -p ~/project_1/results

The -p flag creates all parent directories that don’t exist yet.

View your structure:

find ~/project_1 -type d

Section 2 — Managing Files

Beyond basic navigation, these tools are essential for file management:

Terminal Text Editors

Section 3 — File Permissions

Every file and folder in Linux has permissions controlling read, write, and execute access.

Run ls -l ~/project_1/ to see permissions. Change them with chmod:

Section 4 — Installing Software with apt

Use apt for system-level software, distinct from Python-level packages managed by conda or pip.

Section 5 — Environment Variables

Environment variables are configuration values bash reads at startup.

Section 6 — SSH: Connecting to Remote Machines

SSH lets you control remote machines from your terminal.

  1. Generate SSH key: ssh-keygen -t ed25519 -C "email@example.com"
  2. Connect: ssh username@server_address
  3. Copy key: ssh-copy-id username@server_address (enables password-less login)
  4. Transfer files: Use scp or rsync -avz for efficient directory synchronization

Section 7 — HPC Systems and Job Schedulers

An HPC (High Performance Computing) cluster is not one powerful computer — it’s hundreds or thousands of ordinary computers (“nodes”) wired together, shared by many researchers at once. Because it’s shared, you can’t just run your script the moment you connect. You have to ask for a slice of the cluster’s resources, wait your turn, and let it run there. This section walks through that process in the order you’ll actually encounter it.

📌 A note on schedulers: This tutorial uses SLURM, the most common scheduler at university clusters today. Some clusters instead run PBS/Torque, LSF, or a vendor-specific scheduler — e.g., Japan’s Fugaku supercomputer uses the Fujitsu Technical Computing Suite (pjsub instead of sbatch, pjstat instead of squeue, pjdel instead of scancel). The concepts below are identical across all of them, only the command names change. Check man sbatch or your cluster’s documentation to confirm which one you have.

7.1 — The Basics: Login Nodes, Compute Nodes, and the Scheduler

Before running anything, understand the three pieces you’re working with:

The one rule to remember: if you’re typing a heavy command directly after ssh-ing in, without going through the scheduler, you’re on the login node — stop, and submit it as a job instead (more in 7.6).

7.2 — Submitting Your First Job

A job script describes what you want to run and how much of the cluster it needs. Save this as job.sh:

#!/bin/bash
#SBATCH --job-name=my_job
#SBATCH --output=my_job_%j.out
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=8G
#SBATCH --time=02:00:00

conda activate env_project1
python first_script.py

Each #SBATCH line is a request to the scheduler — here, one task, 4 CPU cores, 8GB memory, and a 2-hour time limit. Go over the time or memory limit and the scheduler kills your job, so estimate generously but not wastefully.

7.3 — Interactive vs. Batch Jobs

sbatch above submits a batch job — it runs unattended, and you only see the result afterward in the .out file. That’s great for long runs, but painful for debugging. For quick tests, request an interactive session instead, which drops you into a live shell on a compute node:

A good habit as a beginner: test your script interactively first, confirm it runs correctly, then submit the same command as a batch job for the full run.

7.4 — Loading Software with Modules (Lmod)

HPC clusters don’t install every version of every tool system-wide — instead, software is made available through modules, managed by a system called Lmod. You load only what your job needs, when it needs it:

⚠️ Loading modules with conflicting dependencies (e.g., two different MPI implementations at once) is a common source of confusing build errors. If something breaks unexpectedly, module purge and reload only what you actually need.

7.5 — Compilers on HPC

Unlike your local machine, a cluster usually offers multiple compiler toolchains side by side — GCC, Intel, and NVIDIA HPC are common — rather than one system compiler, and you pick between them using modules (7.4). This matters for two reasons:

As a beginner, you likely won’t need to touch this directly — but if you ever see a build error mentioning a compiler mismatch, this is why. Check what’s available with module avail gcc or module avail intel.

7.6 — Login Node Etiquette

Worth repeating on its own: running heavy computation on the login node (7.1) slows the cluster down for every other user connected to it at that moment, and most clusters treat it as a policy violation — some will automatically kill offending processes. Always route real work through sbatch, salloc, or srun.

7.7 — Storage Tiers and Quotas

Clusters typically split storage into a few areas, each with different rules:

Check your usage against your limit with quota -s (the exact command varies by site — check your cluster’s documentation).

7.8 — Monitoring and Cancelling Jobs

Once a job is submitted, a few more commands help you manage it:

What’s Next

What You’ve Done:

Further reading:

Next: Part 6 — Git and GitHub for Reproducible Research | Previous: Part 4 — Setting Up VS Code

All Blogs

This site is open source. Improve this page.