Running R in batch mode¶

Overview of the UPPMAX systems¶

Overview of the HPC2N system¶

Overview of the LUNARC system¶

Overview of the NSC system¶

Any longer, resource-intensive, or parallel jobs must be run through a batch script.

The batch system used at UPPMAX, HPC2N, LUNARC, NSC, and PDC (and most other HPC centres in Sweden) is called Slurm.

Slurm is an Open Source job scheduler, which provides three key functions

• Keeps track of available system resources
• Enforces local system resource usage and job scheduling policies
• Manages a job queue, distributing work across resources according to policies

In order to run a batch job, you need to create and submit a SLURM submit file (also called a batch submit file, a batch script, or a job script).

Guides and documentation at: https://docs.hpc2n.umu.se/documentation/batchsystem/intro/ and https://docs.uppmax.uu.se/cluster_guides/slurm/ and https://lunarc-documentation.readthedocs.io/en/latest/manual/manual_intro/ and https://www.nsc.liu.se/support/batch-jobs/introduction/ and https://support.pdc.kth.se/doc/support-docs/run_jobs/job_scheduling/

Workflow¶

• Write a batch script
• Inside the batch script you need to load the modules you need (R and any prerequisites)
• If you are using any own-installed packages, make sure R_LIBS_USER is set (export R_LIBS_USER=/path/to/my/R-packages)
• Ask for resources depending on if it is a parallel job or a serial job, if you need GPUs or not, etc.
• Give the command(s) to your R script
• Submit batch script with sbatch <my-batch-script-for-R.sh>

Common file extensions for batch scripts are .sh or .batch, but they are not necessary. You can choose any name that makes sense to you.

Useful commands to the batch system¶

• Submit job: sbatch <jobscript.sh>
• Get list of your jobs: squeue --me OR squeue -u <username>
• Check on a specific job: scontrol show job <job-id>
• Delete a specific job: scancel <job-id>
• Useful info about a job: sacct -l -j <job-id> | less -S
• Url to a page with info about the job (Kebnekaise only): job-usage <job-id>

.. keypoints::

• The Slurm scheduler handles allocations to the calculation nodes
• Interactive sessions was presented in the previous presentation
• Batch jobs runs without interaction with the user
• A batch script consists of a part with Slurm parameters describing the allocation and a second part describing the actual work within the job, for instance one or several R scripts.
  • Remember to include possible input arguments to the R script in the batch script.

module save preload

module restore preload 

Example R batch scripts¶

Serial code¶

#!/bin/bash -l
#SBATCH -A uppmax2025-Y-ZZZ # Course project id. Change to your own project ID after the course
#SBATCH --time=00:10:00 # Asking for 10 minutes
#SBATCH -n 1 # Asking for 1 core

# Load any modules you need, here R/4.1.1
module load R/4.1.1

# Run your R script (here 'hello.R')
R --no-save --quiet < hello.R

#!/bin/bash
#SBATCH -A hpc2n2025-151 # Change to your own project ID
#SBATCH --time=00:10:00 # Asking for 10 minutes
#SBATCH -n 1 # Asking for 1 core

# Load any modules you need, here R/4.1.2 and prerequisites 
module load GCC/11.2.0  OpenMPI/4.1.1  R/4.1.2

# Run your R script (here 'hello.R')
R --no-save --quiet < hello.R

#!/bin/bash
#SBATCH -A lu2025-2-94 # Change to your own project ID
#SBATCH --time=00:10:00 # Asking for 10 minutes
#SBATCH -n 1 # Asking for 1 core

# Load any modules you need, here R/4.1.2 and prerequisites 
module load GCC/11.3.0  OpenMPI/4.1.4 R/4.2.1

# Run your R script (here 'hello.R')
R --no-save --quiet < hello.R

#!/bin/bash
#SBATCH -A naiss2025-22-934
#SBATCH --time=00:10:00 # Asking for 10 minutes
#SBATCH -n 1 # Asking for 1 core

# Load any modules you need, here R/4.2.2 
module load R/4.2.2-hpc1-gcc-11.3.0-bare 

# Run your R script (here 'hello.R')
R --no-save --quiet < hello.R 

#!/bin/bash -l
#SBATCH -A naiss2025-22-934
#SBATCH --time=00:10:00 # Asking for 10 minutes
#SBATCH -n 1 # Asking for 1 core
#SBATCH -p main

# Load any modules you need, here R/4.4.1
module load PDC/23.12 R/4.4.1-cpeGNU-23.12

# Run your R script (here 'hello.R')
R --no-save --quiet < hello.R             

message <-"Hello World!"
print(message)  

$ sbatch <batch script>

Parallel code¶

foreach and doParallel¶

#!/bin/bash -l
#SBATCH -A uppmax2025-Y-ZZZ
#SBATCH -t 00:10:00
#SBATCH -N 1
#SBATCH -c 4

ml purge > /dev/null 2>&1
ml R_packages/4.1.1

# Batch script to submit the R program parallel_foreach.R 
R -q --slave -f parallel_foreach.R

#!/bin/bash
#SBATCH -A hpc2n2025-151 # Change to your own project ID
#SBATCH -t 00:10:00
#SBATCH -N 1
#SBATCH -c 4

ml purge > /dev/null 2>&1
ml GCC/10.2.0  OpenMPI/4.0.5  R/4.0.4

# Batch script to submit the R program parallel_foreach.R 
R -q --slave -f parallel_foreach.R

#!/bin/bash
# A batch script for running the R program parallel_foreach.R
#SBATCH -A lu2025-2-94 # Change to your own project ID
#SBATCH -t 00:10:00
#SBATCH -N 1
#SBATCH -c 4

ml purge > /dev/null 2>&1
ml GCC/11.3.0  OpenMPI/4.1.4  R/4.2.1

# Batch script to submit the R program parallel_foreach.R
R -q --slave -f parallel_foreach.R

#!/bin/bash
# A batch script for running the R program parallel_foreach.R 
#SBATCH -A naiss2025-22-934
#SBATCH -t 00:10:00
#SBATCH -N 1
#SBATCH -c 4

ml purge > /dev/null 2>&1
ml R/4.2.2-hpc1-gcc-11.3.0-bare 

# Batch script to submit the R program parallel_foreach.R
R -q --slave -f parallel_foreach.R 

#!/bin/bash -l
# A batch script for running the R program parallel_foreach.R
#SBATCH -A naiss2025-22-934 
#SBATCH -t 00:10:00
#SBATCH -N 1
#SBATCH -c 4
#SBATCH -p main

# If you do ml purge you also need to restore the preloaded modules which you should have saved 
# when you logged in. Otherwise comment out the two following lines. 
ml purge > /dev/null 2>&1
ml restore preload
module load PDC/23.12 
module load R/4.4.1-cpeGNU-23.12 

# Batch script to submit the R program parallel_foreach.R
R -q --slave -f parallel_foreach.R

library(parallel)
library(foreach)
library(doParallel)
# Function for calculating PI with no values
calcpi <- function(no) {
  y <- runif(no)
  x <- runif(no)
  z <- sqrt(x^2+y^2)
  length(which(z<=1))*4/length(z)
}
# Detect the number of cores
no_cores <- detectCores() - 1
# Loop to max number of cores
for (n in 1:no_cores) {
  # print how many cores we are using
  print(n)
  # Set start time
  start_time <- Sys.time()
  # Create a cluster
  nproc <- makeCluster(n)
  registerDoParallel(nproc)
  # Create a vector 1000 length with 100 randomizations
  input <- rep(100, 1000)
  # Use foreach on n cores
  registerDoParallel(nproc)
  res <- foreach(i = input, .combine = '+') %dopar%
    calcpi(i)
  # Print the mean of the results
  print(res/length(input))
  # Stop the cluster
  stopCluster(nproc)
  # print end time
  print(Sys.time() - start_time)
  }

$ sbatch <batch script>

Rmpi¶

#!/bin/bash -l
#SBATCH -A uppmax2025-Y-ZZZ
#Asking for 10 min.
#SBATCH -t 00:10:00
#SBATCH -n 8

export OMPI_MCA_mpi_warn_on_fork=0
export OMPI_MCA_btl_openib_allow_ib=1

ml purge > /dev/null 2>&1
ml R_packages/4.1.1
ml openmpi/4.0.3

mpirun -np 1 R CMD BATCH --no-save --no-restore Rmpi.R output.out 

#!/bin/bash
#SBATCH -A hpc2n2025-151 # Change to your own project ID
#Asking for 10 min.
#SBATCH -t 00:10:00
#SBATCH -n 8

export OMPI_MCA_mpi_warn_on_fork=0

ml purge > /dev/null 2>&1
ml GCC/11.2.0  OpenMPI/4.1.1
ml R/4.1.2

mpirun -np 1 Rscript Rmpi.R 

#!/bin/bash
#SBATCH -A lu2025-2-94 # Change to your own project ID
# Asking for 10 min.
#SBATCH -t 00:10:00
#SBATCH -n 8

export OMPI_MCA_mpi_warn_on_fork=0

ml purge > /dev/null 2>&1
ml GCC/11.3.0  OpenMPI/4.1.4
ml R/4.2.1

mpirun -np 1 R CMD BATCH --no-save --no-restore Rmpi.R output.out

#!/bin/bash
#SBATCH -A naiss2025-22-934 
# Asking for 15 min.
#SBATCH -t 00:15:00
#SBATCH -n 8
#SBATCH --exclusive 

ml purge > /dev/null 2>&1
ml R/4.2.2-hpc1-gcc-11.3.0-bare 

srun --mpi=pmix Rscript pbdMPI.R  

#!/bin/bash -l 
#SBATCH -A naiss2025-22-934
# Asking for 10 min.
#SBATCH -t 00:10:00
#SBATCH --nodes 2
#SBATCH --ntasks-per-node=8
#SBATCH -p main
#SBATCH --output=pbdMPI-test_%J.out 

# If you do ml purge you also need to restore the preloaded modules which you should have saved 
# when you logged in. Otherwise comment out the two following lines. 
ml purge > /dev/null 2>&1
ml restore preload
ml PDC/23.12
ml R/4.4.1-cpeGNU-23.12

srun -n 4 Rscript pbdMPI.R

# Load the R MPI package if it is not already loaded.
if (!is.loaded("mpi_initialize")) {
library("Rmpi")
}
print(mpi.universe.size())
ns <- mpi.universe.size() - 1
mpi.spawn.Rslaves(nslaves=ns)
#
# In case R exits unexpectedly, have it automatically clean up
# resources taken up by Rmpi (slaves, memory, etc...)
.Last <- function(){
if (is.loaded("mpi_initialize")){
if (mpi.comm.size(1) > 0){
print("Please use mpi.close.Rslaves() to close slaves.")
mpi.close.Rslaves()
}
print("Please use mpi.quit() to quit R")
.Call("mpi_finalize")
}
}
# Tell all slaves to return a message identifying themselves
mpi.remote.exec(paste("I am",mpi.comm.rank(),"of",mpi.comm.size(),system("hostname",intern=T)))

# Test computations
x <- 5
x <- mpi.remote.exec(rnorm, x)
length(x)
x

# Tell all slaves to close down, and exit the program
mpi.close.Rslaves()

mpi.quit()

library(pbdMPI)

ns <- comm.size()

# Tell all R sessions to return a message identifying themselves
id <- comm.rank()
ns <- comm.size()
host <- system("hostname", intern = TRUE)
comm.cat("I am", id, "on", host, "of", ns, "\n", all.rank = TRUE)

# Test computations
x <- 5
x <- rnorm(x)
comm.print(length(x))
comm.print(x, all.rank = TRUE)

finalize()

$ sbatch <batch script>

Using GPUs in a batch job¶

There are generally either not GPUs on the login nodes or they cannot be accessed for computations. To use them you need to either launch an interactive job or submit a batch job.

UPPMAX only¶

Rackham’s compute nodes do not have GPUs. You need to use Snowy for that.

You need to use this batch command (for x being the number of cards, 1 or 2):

#SBATCH -M snowy
#SBATCH --gres=gpu:x

HPC2N¶

Kebnekaise’s GPU nodes are considered a separate resource, and the regular compute nodes do not have GPUs.

Kebnekaise has a great many different types of GPUs:

• V100 (2 cards/node)
• A40 (8 cards/node)
• A6000 (2 cards/node)
• L40s (2 or 6 cards/node)
• A100 (2 cards/node)
• H100 (4 cards/node)
• MI100 (2 cards/node)

To access them, you need to use this to the batch system:

#SBATCH --gpus=x

where x is the number of GPU cards you want. Above are given how many are on each type, so you can ask for up to that number.

In addition, you need to add this to the batch system:

#SBATCH -C <type>

where type is

• v100
• a40
• a6000
• l40s
• a100
• h100
• mi100

For more information, see HPC2N’s guide to the different parts of the batch system: https://docs.hpc2n.umu.se/documentation/batchsystem/resources/

LUNARC¶

LUNARC has Nvidia A100 GPUs and Nvidia A40 GPUs, but the latter ones are reserved for interactive graphics work on the on-demand system, and Slurm jobs should not be submitted to them.

Thus in order to use the A100 GPUs on Cosmos, add this to your batch script:

• A100 GPUs on AMD nodes:

#SBATCH -p gpua100
#SBATCH --gres=gpu:1

These nodes are configured as exclusive access and will not be shared between users. User projects will be charged for the entire node (48 cores). A job on a node will also have access to all memory on the node.

• A100 GPUs on Intel nodes:

#SBATCH -p gpua100i
#SBATCH --gres=gpu:<number>

where is 1 or 2 (Two of the nodes have 1 GPU and two have 2 GPUs).

NSC¶

Tetralith has Nvidia T4 GPUs. In order to access them, add this to your batch script or interactive job:

#SBATCH -n 1
#SBATCH -c 32
#SBATCH --gpus-per-task=1

PDC¶

Dardel has AMD AMD Instinct™ MI250X GPU chips. In order to access them, add this to your batch script or interactive job:

#SBATCH -N 1
#SBATCH --ntasks-per-node=1
#SBATCH -p gpu  

Example batch script¶

#!/bin/bash -l 
#SBATCH -A uppmax2025-Y-ZZZ 
#Asking for runtime: hours, minutes, seconds. At most 1 week
#SBATCH -t HHH:MM:SS
#SBATCH --exclusive
#SBATCH -p node
#SBATCH -N 1
#SBATCH -M snowy
#SBATCH --gpus=1
#SBATCH --gpus-per-node=1
#Writing output and error files
#SBATCH --output=output%J.out
#SBATCH --error=error%J.error

ml purge > /dev/null 2>&1
ml R/4.1.1 R_packages/4.1.1

R --no-save --no-restore -f MY-R-GPU-SCRIPT.R

#!/bin/bash
#SBATCH -A hpc2n2025-151 # Change to your own project ID
#Asking for runtime: hours, minutes, seconds. At most 1 week
#SBATCH -t HHH:MM:SS
#Ask for GPU resources. You pick type as one of the ones shown above 
#x is how many cards you want, at most as many as shown above 
#SBATCH --gpus:x
#SBATCH -C type
#Writing output and error files
#SBATCH --output=output%J.out
#SBATCH --error=error%J.error

ml purge > /dev/null 2>&1
#R version 4.0.4 is the only one compiled for CUDA 
ml GCC/10.2.0  CUDA/11.1.1 OpenMPI/4.0.5
ml R/4.0.4

R --no-save --no-restore -f MY-R-GPU-SCRIPT.R

#!/bin/bash
# Remember to change this to your own project ID after the course!
#SBATCH -A lu2025-2-94
# Asking for runtime: hours, minutes, seconds. At most 1 week
#SBATCH --time=HHH:MM:SS
# Ask for GPU resources - x is how many cards, 1 or 2 
#SBATCH -p gpua100
#SBATCH --gres=gpu:x

# Remove any loaded modules and load the ones we need
module purge  > /dev/null 2>&1
module load GCC/11.3.0  OpenMPI/4.1.4 R/4.2.1 CUDA/12.1.1 

R --no-save --no-restore -f MY-R-GPU-SCRIPT.R

#!/bin/bash
# Remember to change this to your own project ID after the course!
#SBATCH -A naiss2025-22-934
# Asking for runtime: hours, minutes, seconds. At most 1 week
#SBATCH --time=HHH:MM:SS
# Ask for resources, including GPU resources
#SBATCH -n 1
#SBATCH -c 32
#SBATCH --gpus-per-task=1

# Remove any loaded modules and load the ones we need
module purge  > /dev/null 2>&1
module load R/4.4.0-hpc1-gcc-11.3.0-bare 

R --no-save --no-restore -f MY-R-GPU-SCRIPT.R

#!/bin/bash -l 
# Remember to change this to your own project ID after the course!
#SBATCH -A naiss2025-22-934   
# Asking for runtime: hours, minutes, seconds. At most 1 week
#SBATCH --time=HHH:MM:SS
# Ask for resources, including GPU resources
#SBATCH -N 1
#SBATCH --ntasks-per-node=1
#SBATCH -p gpu 

module load PDC/23.12 R/4.4.1-cpeGNU-23.12 
module load rocm/5.7.0
#module load craype-accel-amd-gfx90a 
#module load cpeGNU/23.12
R --no-save --no-restore -f MY-R-GPU-SCRIPT.R

Exercises¶

#!/bin/bash -l
#SBATCH -A uppmax2025-Y-ZZZ # Change to your own after the course
#SBATCH --time=00:10:00 # Asking for 10 minutes
#SBATCH -n 1 # Asking for 1 core

# Load any modules you need, here for R/4.1.1
module load R/4.1.1

# Run your R script 
Rscript add2.R 2 3 

#!/bin/bash
#SBATCH -A hpc2n2025-151 # Change to your own project ID
#SBATCH --time=00:10:00 # Asking for 10 minutes
#SBATCH -n 1 # Asking for 1 core

# Load any modules you need, here for R/4.1.2
module load GCC/11.2.0  OpenMPI/4.1.1 R/4.1.2

# Run your R script 
Rscript add2.R 2 3 

#!/bin/bash
#SBATCH -A lu2025-2-94 # Change to your own project ID
#SBATCH --time=00:10:00 # Asking for 10 minutes
#SBATCH -n 1 # Asking for 1 core

# Load any modules you need, here for R/4.2.1
module load GCC/11.3.0  OpenMPI/4.1.4 R/4.2.1

# Run your R script 
Rscript add2.R 2 3 

#!/bin/bash
#SBATCH -A naiss2025-22-934 
#SBATCH --time=00:10:00 # Asking for 10 minutes
#SBATCH -n 1 # Asking for 1 core

# Load any modules you need, here for R/4.2.2 
module load R/4.2.2-hpc1-gcc-11.3.0-bare 

# Run your R script 
Rscript add2.R 2 3 

#!/bin/bash
#SBATCH -A naiss2025-22-934 
#SBATCH --time=00:10:00 # Asking for 10 minutes
#SBATCH -n 1 # Asking for 1 core
#SBATCH -p main 

# Load any modules you need, here for R/4.4.1 
module load PDC/23.12 R/4.4.1-cpeGNU-23.12 

# Run your R script 
Rscript add2.R 2 3