Abhigyan Chakraborty

Linux Essentials for HPC and Remote Servers

Phase 2: The Research Infrastructure — Part 1

(Part 5 of series Blueprint for a Modern Research Computing Environment)

Follow me :

Quick Summary

Unlike previous parts, this article serves as a map rather than a build tutorial. It covers directory organization, file management, permissions, installing system software with apt, environment variables, SSH, and the basics of HPC clusters and SLURM job submission — the concepts researchers need once they move beyond their local machine to remote servers.

Objective

In previous parts, you built a concrete environment on your local machine. Here, the goal is different: this is a map. As a researcher, you’ll eventually work on remote servers and university HPC (High Performance Computing) clusters. This article provides the foundational knowledge needed to navigate these environments confidently.

By the end, you’ll have:

An understanding of directory structures, file management, and permissions.
Knowledge of system-level software installation using apt.
Clarity on environment variables and how to configure them.
Familiarity with remote access via SSH and job scheduling via SLURM.

Content

💡 Getting Unstuck (Expand for AI Troubleshooting Prompts)

AI tools (like Claude, Grok, Gemini, or ChatGPT) give the best troubleshooting advice when they have the exact text of the tutorial. To ensure the AI understands what you are trying to build, we will give it the actual file.

1. Download the tutorial file

Open this link in a new tab: Part5.md on GitHub
Look near the top-right corner of the text box on that page.
Click the Download raw file button (it looks like a downward arrow ⬇️).

2. Upload it to your AI

Open your AI tool (ChatGPT, Claude, Gemini, etc.).
Click the paperclip (📎) or plus (➕) icon next to the text box to upload the file (or simply drag and drop Part5.md into the chat).

3. Ask for Help Copy and paste this exact prompt into the chat along with your file:

I have attached the markdown file for the tutorial I am following. Please read it so you understand the specific environment I am trying to build.

I need help with the following:

Step [X]: [paste exact step text from the blog]

Command I ran: [paste exact command]

What happened: [paste full output/error — if the terminal was truly blank, say so explicitly]

Please help me troubleshoot and fix this error. You can use your general knowledge to solve the problem, but your solution MUST align with the architecture and tools taught in the attached file. Do not suggest alternative setups that contradict the tutorial. Once fixed, tell me what to do next in the article.

To go deeper on a step before you run a command:

I have attached the Part5.md file. Look at Step [X] and explain exactly what the command does and why we are doing it before I run it.

Think of this series as the roadmap and your AI assistant as your learning companion.

Prerequisites

Comfortable with the terminal (Part 2)
~/project_1 with env_project1 set up (Part 3)

Section 1 — Creating and Organizing Directories

Create nested directories in one command:

mkdir -p ~/project_1/data/raw
mkdir -p ~/project_1/data/processed
mkdir -p ~/project_1/scripts
mkdir -p ~/project_1/results

The -p flag creates all parent directories that don’t exist yet.

View your structure:

find ~/project_1 -type d

Section 2 — Managing Files

Beyond basic navigation, these tools are essential for file management:

Delete a file: rm file.txt
Delete a folder: rm -rf foldername/ (⚠️ This is permanent; there is no recycle bin)
View file contents: cat file.txt or less file.txt (press q to exit)
Check disk usage: du -sh ~/project_1/

Terminal Text Editors

nano: Simple, opens files directly in the terminal; ideal for quick config edits.
vim: More powerful but has a steep learning curve; useful for remote servers.

Section 3 — File Permissions

Every file and folder in Linux has permissions controlling read, write, and execute access.

Run ls -l ~/project_1/ to see permissions. Change them with chmod:

Make a file executable: chmod +x first_script.py
Restrict file access: chmod 600 ~/.ssh/id_rsa

Section 4 — Installing Software with `apt`

Use apt for system-level software, distinct from Python-level packages managed by conda or pip.

Update repositories: sudo apt update
Install Git: sudo apt install git
Install compilers: sudo apt install gcc g++ gfortran

Section 5 — Environment Variables

Environment variables are configuration values bash reads at startup.

View variables: env or echo $PATH
Create a variable: export MY_DATA="/mnt/d/datasets"
Make permanent: Add the export line to ~/.bashrc and run source ~/.bashrc

Section 6 — SSH: Connecting to Remote Machines

SSH lets you control remote machines from your terminal.

Generate SSH key: ssh-keygen -t ed25519 -C "email@example.com"
Connect: ssh username@server_address
Copy key: ssh-copy-id username@server_address (enables password-less login)
Transfer files: Use scp or rsync -avz for efficient directory synchronization

Section 7 — HPC Systems and SLURM

HPC clusters connect hundreds of computers to run massive jobs. Unlike local machines, you submit these jobs to a queue managed by a scheduler like SLURM.

A basic SLURM job script (job.sh):

#!/bin/bash
#SBATCH --job-name=my_job
#SBATCH --output=my_job_%j.out
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=8G
#SBATCH --time=02:00:00

conda activate env_project1
python first_script.py

Submit: sbatch job.sh
Check status: squeue -u username

On HPC, use the module system to load software: module avail, module load python/3.11, module list.

What’s Next

What You’ve Done:

Organized directories and managed files
Understood permissions, system software installation, and environment variables
Learned SSH for remote connectivity and SLURM for HPC job submission

Further reading:

Next: Part 6 — Git and GitHub for Reproducible Research | Previous: Part 4 — Setting Up VS Code

All Blogs

This site is open source. Improve this page.