Saluki Compute

Slurm GPU and Jupyter User Guide

Use Slurm to allocate the CPUs, memory, and GPUs needed by your work on the Rocky/Alma Linux labservers. You can launch Jupyter Lab in a browser, work in an interactive terminal, or submit a script that runs without an open SSH connection. Jobs wait in the queue until their requested resources are available.

Updated: October 9, 2026. Start with the scripts in /tutorials on the server you are using. Software, GPU models, and resource limits differ between servers. Commands below have no shell prompt to remove before copying. Replace uppercase placeholders such as JOBID with your own values; keep shell variables such as $USER unchanged.

Jupyter quick start

After connecting to your assigned server, create a working directory and copy the current launcher:

mkdir -p ~/slurmjob
cd ~/slurmjob
cp /tutorials/run_jupyter_shared.sbatch .
sbatch --gres=gpu:1 --cpus-per-task=2 --mem=4G --time=01:00:00 \
  run_jupyter_shared.sbatch

This example requests one GPU, two CPUs, 4G of memory, and one hour. Increase resources for work that needs them, within the server’s limits. Submission prints a job number. For example, this is output, not a command:

Submitted batch job 12345

Check the queue:

squeue -u "$USER"

Once your job starts, follow its output. Replace JOBID with your job number. The output file is created when the job starts, so it may not exist while the job is pending:

tail -f slurm-JOBID.out

Press Ctrl+C to stop following the output; this does not cancel the job.

Wait for the job to start. Its output prints Jupyter logs: and then Access details: followed by a private file path. Read the exact file printed by your job, replacing ACCESS_FILE below:

cat "ACCESS_FILE"

Open its complete HTTPS URL, including the token, in a browser connected to SIU’s campus network or VPN. If prompted for a token, use the one in that URL. These notebook sessions use a self-signed certificate; the browser may ask you to confirm an exception for the notebook address. Keep the URL and token private because they provide access to your session.

Save work in the notebook’s /workspace directory. Closing the browser leaves the job running. When finished, release its resources:

scancel JOBID

Jupyter quick start
Connect and check resources
Choose how to run your work
Use Jupyter effectively
Run terminal and batch applications
Monitor and stop jobs
Store files and save progress
Plan around maintenance
Advanced environments and containers
Schedule future work and request priority help
Tutorials and support

Connect and check resources

Your DawgTag account must be granted access before you can connect. Use your assigned server hostname or IP address and replace both placeholders:

ssh YOUR_USERNAME@SERVER

After login, identify the server and inspect its Slurm resources:

hostname
sinfo -N -o "%N %c %m %G"

The columns show the Slurm node name, configured CPUs, memory in MiB, and GPU resources, called GRES. The Slurm node may be named localhost; hostname identifies the server you reached. These values are configured totals, not a count of resources currently free. Use squeue to see running and pending jobs.

Use $HOME for your home directory rather than assuming a particular storage path. Check available home storage with:

df -h "$HOME"
du -sh "$HOME"

If SSH access has not been arranged, contact the administrator listed under Tutorials and support.

Back to contents

Choose how to run your work

Jupyter: Submit the provided launcher with sbatch and use the notebook in your browser.
Interactive terminal: Request a shell with srun, then run your application inside that allocated shell.
Batch program: Put commands in an .sbatch file and submit it with sbatch. You can disconnect while it runs.

For GPU work, --gres=gpu:1 requests one GPU. Run the application inside the resulting allocation. Loading a CUDA module makes the toolkit available; it does not allocate hardware or give a separate SSH shell the job’s GPU access. CPU-only jobs do not receive GPU access.

Understand GPU requests

sinfo -N -o "%N %G"

gpu:2 in this output means two GPUs are configured, not a GPU model named “2.” Requesting --gres=gpu:1 asks for one GPU. If Slurm reports gpu:TYPE:COUNT, use that exact registered type in --gres=gpu:TYPE:1 when you need a particular model. Cards shown by nvidia-smi are not necessarily all registered with Slurm.

Keep Slurm’s CUDA_VISIBLE_DEVICES unchanged. A one-GPU job normally uses cuda:0, even when the physical card has another host display number. You do not need to select a physical GPU index yourself. See the CS-545360 companion for that server’s H100 and Blackwell choices.

Back to contents

Use Jupyter effectively

Resource requests and job lifetime

The copied launcher selects a shared image installed on that server. You do not need to load a host CUDA module to start it; the container includes its own libraries. Keep the launcher’s startup and cleanup commands intact.

Edit resource directives in your copy, or override them when submitting. These are examples of directives, not a complete replacement launcher:

#SBATCH --gres=gpu:1
#SBATCH --cpus-per-task=2
#SBATCH --mem=4G
#SBATCH --time=01:00:00

The time limit begins when the job starts. Cancelling the job or reaching the limit stops its container and browser access. Save notebooks and checkpoints before then. The launcher assigns and releases the browser port automatically; if it reports no assigned port, send the administrator the job ID and error.

Workspace and private logs

The default workspace is /scratch/SHORT_USERNAME/shared_jupyter_dir when /scratch exists, or $HOME/shared_jupyter_dir otherwise. The job output identifies the actual host path mounted as /workspace inside the notebook. Files there follow the host directory’s retention policy.

To choose your home directory for the next submission:

export JUPYTER_WORKSPACE_BASE=home
sbatch run_jupyter_shared.sbatch

JUPYTER_USE_JOB_DIR=true selects a separate directory for each job. JUPYTER_WORKSPACE_DIR can select an explicit path. Review the copied script before changing these settings.

Each attempt has its own private jupyter-logs/JOBID.RANDOM/ directory. The Access details: path points to its access_jupyter.log; jupyter.log contains startup output and errors. Use the latest directory printed after a requeue. Package installation in postStart.sh can delay startup. Read the current log before submitting duplicate jobs.

Select the notebook environment

Choose the kernel labeled Data Science for env_ds, which provides tools such as NumPy, pandas, scikit-learn, and plotting libraries. Choose Deep Learning for env_dl, which includes PyTorch and other learning tools. Kernel labels and package versions vary by image.

The base environment is primarily for Jupyter itself. From a terminal inside the container, check environments and run a PyTorch script with:

conda env list
conda run -n env_dl python your_script.py

Replace your_script.py with your program. Verify its required packages before a long run.

Back to contents

Run terminal and batch applications

Start an interactive GPU shell

srun --partition=debug --gres=gpu:1 --cpus-per-task=2 \
  --mem=4G --time=01:00:00 --pty bash -l

Wait for the allocated shell. For native host CUDA, run the following inside it, followed by your application:

source /etc/profile.d/modules.sh
module avail cuda
module load cuda
nvcc --version
printf 'Job %s; CUDA devices: %s\n' "$SLURM_JOB_ID" "$CUDA_VISIBLE_DEVICES"

Use a version shown by module avail cuda if you need to pin the toolkit, such as module load cuda/13.2. The module supplies the host compiler and libraries; it does not install Python frameworks or alter a container’s toolkit. Type exit when finished to release the interactive allocation. For CPU-only work, omit --gres=gpu:1.

Submit a native CUDA batch script

Save this as native_cuda.sbatch. Replace ./my_cuda_application with your application command, and adjust its resources:

#!/bin/bash
#SBATCH --job-name=native_cuda
#SBATCH --partition=debug
#SBATCH --gres=gpu:1
#SBATCH --cpus-per-task=2
#SBATCH --mem=4G
#SBATCH --time=01:00:00
set -euo pipefail
source /etc/profile.d/modules.sh
module load cuda
./my_cuda_application

Submit from the directory containing your application and script:

sbatch native_cuda.sbatch

Running a job script with bash does not create a Slurm allocation. Use sbatch, then monitor the returned job number.

Back to contents

Monitor and stop jobs

Check your queue and inspect a job, replacing JOBID:

squeue -u "$USER"
scontrol show job JOBID

Common states are PD for pending, R for running, and CG for completing. Pending jobs have requested resources, not an active allocation. Their execution time is normally 0:00; that value does not tell you how long they have waited. A displayed start time for a pending job is an estimate.

Review accounting after a job finishes:

sacct -u "$USER" -S today --format=JobID,JobName,State,ExitCode,Elapsed
sacct -j JOBID --format=JobID,State,TotalCPU,MaxRSS,AllocTRES
seff JOBID

Keep step rows such as .batch and .0 when inspecting memory use. Very short jobs may end before a memory sample is collected. AllocTRES describes allocated resources; it does not measure GPU utilization or GPU memory use.

Run nvidia-smi inside your GPU job or use the launcher’s GPU logs. ENABLE_GPU_LOGGING=true enables logging; SEPARATE_LOGS=true writes separate GPU files. Current launchers use gpu_usage_log_gpu*.out or gpu_usage_log.out, plus gpu_summary.out after cleanup, in the private job log directory. GPU query errors go to gpu-errors.log.

Choose memory requests using observed application needs. The notebook container’s /dev/shm maximum is 80% of the requested job memory; it is not preallocated RAM. Shared-memory pages and other process allocations together must fit the job budget. Host free -h output is not a Slurm reservation.

Release a finished notebook, unwanted pending job, or running job you no longer need:

scancel JOBID

Back to contents

Store files and save progress

Keep important code, datasets, environment definitions, and checkpoints in appropriate persistent host storage, with a separate backup. Save notebook work through /workspace. Files saved only in a disposable container’s root filesystem disappear when that container is removed.

$HOME is outside the scratch cleanup job described below, but remains subject to storage limits and site policy. Ask the administrator about large datasets, quotas, or dedicated storage. Neither a home directory nor scratch storage is a backup.

Scratch retention on CS-541866

On CS-541866, /scratch and /scratch2 are temporary. Cleanup runs Sundays at 3:30 a.m. Central with a 30-day modification-time cutoff. Reading files or running software does not refresh that time. The Monday 9 a.m. warning scan does not grant another seven days or guarantee an individual warning.

Review .scratch_cleanup_notice.txt and .scratch_cleanup_deleted_*.txt in your scratch directory. Other servers may have different policies; check before relying on their scratch storage.

To refresh a specific working tree you still need, replace PROJECT_DIRECTORY with its directory name:

find /scratch2/"$USER"/PROJECT_DIRECTORY -exec touch -h -c -- {} +

This refreshes files, directories, and symbolic links, including hidden entries. It does not traverse a linked directory or refresh its target elsewhere. Refresh separately stored targets when needed and read any errors. Touching only a directory does not refresh its contents. Do not exclude all .pth files when maintaining a Python environment; some configure that environment. Refreshing timestamps is not a permanent exemption, a backup, or a way to restore deleted files.

The cleanup rechecks candidates, retains links while the link or target is recent, and removes only expired empty directories. It defers deletion when the account has active Slurm jobs or their status cannot be verified. These checks do not make scratch permanent storage.

Save restartable checkpoints

Save progress periodically during long jobs, not only at completion. Test that your application can reload a checkpoint before committing to a long run. The server provides checkpoint guidance at:

/tutorials/GPU_User_Guide/Saving_Progress.md

Inside Jupyter, time- or epoch-based PyTorch and TensorFlow examples are in /workspace/examples/deep_learning/periodic_checkpointing after the launcher extracts the tutorial notebooks.

Keep checkpoints on mounted host storage and copy important results to your backup location.

Back to contents

Plan around maintenance

Before a long run, check the server’s current plan:

smartupdate-maintenance status

When advance draining is enabled, Slurm stops starting new jobs before the scheduled update. Running jobs can finish, while queued jobs remain pending. If eager updates are enabled and all active Slurm jobs finish, maintenance can begin early, including outside usual maintenance hours, after a ten-minute terminal warning. At the scheduled cutoff, remaining jobs may be cancelled. A reboot follows when needed after update checks pass.

The warning is not a guaranteed browser notification. Keep notebooks saved and checkpoints current. Everyone can view the schedule, but only configured accounts can postpone it. Permission to postpone does not include permission to cancel everyone’s jobs and force an update. Read the local instructions:

/tutorials/smart-update-tool/Smart-Update-End-User-Guide.md

Back to contents

Advanced environments and containers

Create a temporary Conda environment

Open a terminal inside Jupyter. Create your own environment rather than changing the shared data science or deep learning environments:

conda create -n my_env python=3.10
source /opt/conda/bin/activate my_env
conda install ipykernel
python -m ipykernel install --user --name=my_env --display-name 'Python 3.10 my_env'

Install the packages your project needs and refresh Jupyter to find the new kernel. Environments created in the container’s temporary layer disappear at job completion. Keep the commands or environment definition needed to recreate them in host storage.

Automate setup with postStart.sh

The launcher can run an executable postStart.sh from your submission directory before starting Jupyter. Create that file in your working directory:

mkdir -p ~/slurmjob
nano ~/slurmjob/postStart.sh

For example, save the following contents to create and register a temporary environment. Adjust the packages for your project:

#!/bin/bash
set -e
source /opt/conda/etc/profile.d/conda.sh

if ! conda info --envs | grep -q '^my_env '; then
    conda create -n my_env python=3.10 -y
    conda install -n my_env -y pandas scikit-learn ipykernel -c conda-forge
fi
conda activate my_env
python -m ipykernel install --user --name=my_env \
    --display-name 'Python 3.10 my_env'

Make it executable and submit from the directory containing the script:

cd ~/slurmjob
chmod u+x postStart.sh
sbatch run_jupyter_shared.sbatch

Installation delays notebook startup. Inspect the job’s private jupyter.log if setup fails. Keep postStart.sh as a reproducible recipe rather than relying on a previous container’s installed files.

Run a custom Podman image

The Jupyter launchers already handle Podman. For a custom container, first enter a GPU allocation with srun, or put the commands in a submitted batch script. Initialize your rootless mapping before listing available images:

sudo /usr/local/sbin/allocate_uid_gid.sh
podman images --format '{{.Repository}}:{{.Tag}}'

Replace IMAGE:TAG with an installed image compatible with your application and GPU. This example checks GPU visibility inside that image:

podman run --rm --pull=never \
  --cgroup-manager=cgroupfs --cgroups=disabled --cgroupns=host \
  --device nvidia.com/gpu=all --security-opt=label=disable \
  -e CUDA_VISIBLE_DEVICES -e CUDA_DEVICE_ORDER=PCI_BUS_ID \
  --entrypoint nvidia-smi IMAGE:TAG

Keep these settings to retain the Slurm allocation’s device limits, and run Podman as your normal account. The approved mapping helper is the command that uses sudo; do not replace the container command with sudo podman. For a CPU-only container, use a CPU-only allocation and omit the GPU device and CUDA environment options. Image customization instructions are on the server:

/tutorials/Dockerfile-examples/docs/Create_Permanent_Image_README.md

Use Flox for a project environment

Flox is installed but does not activate an environment automatically. For a project outside the shared Jupyter workflow:

command -v flox
flox --version
mkdir my-flox-project
cd my-flox-project
flox init
flox search python
flox install python
flox activate
flox list

Later, return to the project directory and run flox activate again. Use flox --help for more commands. An environment manager does not allocate compute resources; run your workloads through Slurm.

Keep an interactive terminal with Screen

Start Screen before requesting an interactive allocation:

screen -S my_screen_session
srun --partition=debug --gres=gpu:1 --cpus-per-task=2 \
  --mem=4G --time=01:00:00 --pty bash -l

Run the application in the shell opened by srun. To detach, press Ctrl+A, then D. Reconnect to the server and reattach with:

screen -ls
screen -r my_screen_session

Screen keeps the terminal session available across disconnects; it does not extend the job’s time limit or create its own GPU allocation. Type exit to leave the allocated shell when finished, then exit again to close Screen.

Back to contents

Schedule future work and request priority help

--begin sets the earliest time a job becomes eligible to start. It does not reserve resources or guarantee that time:

sbatch --begin=now+2days run_your_job.sbatch
sbatch --begin=2026-11-17T08:00:00 run_your_job.sbatch

Replace the script name and date as appropriate, using the server’s local timezone. --time limits duration from the actual start; it is not a fixed clock-time booking.

For a reserved session, ask the administrator to arrange a reservation for your account. Agree its resources, date, duration, and name. Use the name they provide in your submission, for example:

sbatch --reservation=research_session --partition=debug \
  --gres=gpu:1 --time=01:50:00 run_your_job.sbatch

Your job duration must fit within the reservation. A whole-node reservation on a single-node server blocks other users from that node’s resources during the booking. Coordinate reservations with maintenance: SmartUpdate does not automatically avoid future Slurm reservations.

Regular users cannot arbitrarily raise job priority. Contact the administrator about a deadline or scheduling problem rather than trying to override queue policy. Arrangements depend on resource availability and other users’ needs.

Back to contents

Tutorials and support

Copy examples into a working directory you own before editing or submitting them. /tutorials includes the Jupyter launchers, Dockerfile examples, and checkpoint guidance. The GPU-monitoring Jupyter launcher also copies notebook examples into /workspace/examples, including pandas, scikit-learn, plotting, PyTorch, TensorFlow, and periodic checkpointing examples.

Choose examples suited to the installed notebook environment. Save changes in your mounted workspace and keep important results backed up. Ask the administrator if your project needs a different shared image or additional software.

For help, contact School of Computing system administration at mabarkdoll@siu.edu. Include the server hostname, job ID, command or script used, and relevant error text. For a GPU problem, include the observed GPU UUID if available. Keep notebook access tokens out of shared logs and messages.

Additional reference material:

Slurm GPU and Jupyter User Guide

Jupyter quick start

On this page

Connect and check resources

Choose how to run your work

Understand GPU requests

Use Jupyter effectively

Resource requests and job lifetime

Workspace and private logs

Select the notebook environment

Run terminal and batch applications

Start an interactive GPU shell

Submit a native CUDA batch script

Monitor and stop jobs

Store files and save progress

Scratch retention on CS-541866

Save restartable checkpoints

Plan around maintenance

Advanced environments and containers

Create a temporary Conda environment

Automate setup with postStart.sh

Run a custom Podman image

Use Flox for a project environment

Keep an interactive terminal with Screen

Schedule future work and request priority help

Tutorials and support

Contact Us

Contact Us