You SSH into the server, activate your environment, and run the training script that renders a few evaluation videos. Everything worked perfectly on your laptop. Here, you get something like this:
mujoco.FatalError: gladLoadGL error
or, if you are slightly luckier:
RuntimeError: Failed to initialize EGL display
No GPU is broken. Your driver is fine — nvidia-smi prints a beautiful table. The problem is that MuJoCo wants to render headlessly, and the one small piece of the driver it needs for that is missing from the system. The annoying part is that fixing it normally requires sudo, which you probably do not have on a shared machine.
Good news: you can fix it from your own home directory. This post explains why the error happens and walks through the workaround.
You type pip install flash-attn. You hit Enter. You wait. And wait. Your fan spins up like a jet engine. 10 minutes later: compilation error. Congratulations, you've experienced one of the most classic rites of passage in the ML world.
This guide explains what's going on, why some packages are a nightmare to install, and how to sidestep the pain with prebuilt wheels.
This tutorial assumes you already know the basics of PyTorch and how to train a model.
If you've been training deep learning models on a single GPU and wondering how to scale up, or if you've heard terms like "data parallelism" and "AllReduce" thrown around without really understanding what they mean - this series is for you. We're going to build your understanding from the ground up, starting with the absolute basics of how multiple GPUs can work together, all the way to implementing production-ready distributed training systems.
This first article lays the foundation. We'll understand why we need multiple GPUs, how they actually communicate with each other, and what happens under the hood when you run distributed training. By the end, you'll have the mental model needed to understand everything that comes next in this series.
If you're only working on the BASIC Lab server, Slurm might not be necessary yet. However, if you plan to use NCHC resources, then learning Slurm is a must. All NCHC clusters are managed through Slurm.
So you've finally gotten access to NCHC's fancy compute cluster. No more waiting three days using slow and outdated GPUs to finish training that model. You're ready for the big leagues!
You SSH in, navigate to your directory, and type the command you've been using on your local machine for months: python my_awesome_script.py
And then... someone taps you on the shoulder (virtually or literally) and says, "Oh no, you can't just run things directly here. You need to submit it through Slurm."
When using the provided server, everything including the driver and CUDA toolkit is already installed, so you might not need to worry about these details initially. However, I strongly encourage you to understand these concepts because you might one day need to maintain your own server (though hopefully you won't have to).
Back in the day, I always wondered why we could run PyTorch code on our local machine without a GPU, but when it came to compiling or training local library, we suddenly needed CUDA toolkit. What's going on under the hood?
In this article, we’ll break down the mystery behind CUDA, cuDNN, and all the other buzzwords. By the end, you’ll have a clearer (and hopefully less intimidating) understanding of how they all fit together.
Code Style Note: The demonstration code examples in this post are intentionally compact for readability. In production code, you should follow PEP 8 style guidelines with proper spacing, line breaks, and formatting.
This guide shows you how to automatically format your Python code using Black and isort, so your project stays clean, consistent, and easy to read (even at 3 a.m. after too much coffee).
Black: An opinionated Python code formatter that auto-formats your code to follow PEP 8 style (so you can stop arguing about whitespace with your teammates).
isort: Automatically sorts and organizes your imports into neat, logical sections.
This post may be updated as better alternatives to uv emerge.
uv is a super-fast Python package installer and resolver, built in Rust. Think of it as a modern, turbocharged alternative to pip and other Python package managers.