HPC sample job scripts

Jobs are submitted to SLURM as a job script, which lists the commands to run and gives instructions to SLURM on how to treat the job.  These SLURM instructions are lines beginning #SBATCH. All #SBATCH lines must be at the top of your scripts, before any other commands, or they will be ignored. 

To submit a job to SLURM, adapt one of the example job scripts below, save your job script with a suitable name and type

sbatch myjobscriptname

The sbatch man page has information about the #SBATCH options available.  Always try to specify your job's run-time and/or memory needs, rather than accepting the defaults or the maximum allowed.  If you ask for too much, SLURM may take longer to schedule your jobs and will overstate your usage, affecting the priority it gives to your future jobs.  However, if you ask for too little, your job may crash or be stopped. 

Serial (non-parallel) job

#!/bin/bash
# Example SLURM job script for serial (non-parallel) jobs
#
# SLURM defaults to the directory you were working in when you submitted the job.
# Output files are also put in this directory. To set a different working directory add:
#
#SBATCH –-workdir=/my_path_to/my_directory
#
# Tell SLURM if you want to be emailed when your job starts, ends, etc.
#
#SBATCH --mail-type=ALL
#

echo Starting job in directory `pwd`
sleep 60
echo Finishing job

Job array

If you need to submit a set of very similar jobs, use a job array. Job arrays make use of shell variables, including ${SLURM_ARRAY_TASK_ID} that identifies each job in the array. You can also use %A to refer to the job ID and %a to refer to the array index.  Array indices can be set as needed, e.g. #SBATCH --array=1-9:2,12,19 would run an array with elements 1,3,5,7,9,12 and 19.

#!/bin/bash
# Example SLURM job script for job arrays
#
#
# This job runs a command called myprog and takes input from a set of files called 
# nnn.input, where nnn is the array task IDs
#
# set up a job array with tasks numbered 1 to 100.
#SBATCH --array=1-100
#
# give the array a single job name
#SBATCH -J myarrayjobname
#
# Standard output is saved in this file
#SBATCH -o myjob_%A_%a.out
#
# Standard error messages are saved in this file
#SBATCH -e myjob_%A_%a.err
#
# request one core per task
#SBATCH -c 1
#
# request a run-time of 10 minutes
#SBATCH -t 0-0:10:00
#
#
# Tell SLURM if you want to be emailed when your job starts, ends, etc.
# Currently mail can only be sent to addresses @ncl.ac.uk
#SBATCH --mail-type=ALL
#
# use the task ID to locate the input file for the task.
myprog ./${SLURM_ARRAY_TASK_ID}.input

Multi-threaded/OpenMP job

#!/bin/bash
#example SLURM job script for an OpenMP job
#
# SLURM defaults to the directory you were working in when you submitted the job.
# Output files are also put in this directory. To set a different working directory add:
#
#SBATCH –-workdir=/nobackup/my_directory
#
# Tell SLURM if you want to be emailed when your job starts, ends, etc.
# Currently mail can only be sent to addresses @ncl.ac.uk
#
#SBATCH --mail-type=ALL
#
# cores per task
#SBATCH -c 22
#
# number of nodes
#SBATCH -N 1
#

module load OpenMP

#
# set the $OMP_NUM_THREADS variable
ompthreads=$SLURM_JOB_CPUS_PER_NODE
export OMP_NUM_THREADS=$ompthreads
#

./my_omp_program

MPI job

#!/bin/bash
# example MPI job script for SLURM
#
# SLURM defaults to the directory you were working in when you submitted the job.
# Output files are also put in this directory. To set a different working directory add:
#
#SBATCH –-workdir=/nobackup/my_directory
#
# Tell SLURM if you want to be emailed when your job starts, ends, etc.
# Currently mail can only be sent to addresses @ncl.ac.uk
#
#SBATCH --mail-type=ALL
#
# number of tasks to use
#SBATCH --ntasks=88
#
# use Intel programming tools
module load intel
#
# SLURM recommend using srun instead of mpirun for better job control.
srun ./mpi_program

 Hybrid OpenMP/MPI job

#!/bin/bash
# example MPI+OpenMP job script for SLURM
#
# SLURM defaults to the directory you were working in when you submitted the job.
# Output files are also put in this directory. To set a different working directory add:
#
#SBATCH –-workdir=/nobackup/my_directory
#
# Tell SLURM if you want to be emailed when your job starts, ends, etc.
# Currently mail can only be sent to addresses @ncl.ac.uk
#
#SBATCH --mail-type=ALL
#
# This example has 4 MPI tasks, each with 22 cores
#
# number of tasks
#SBATCH --ntasks=4
#
# number of cores per task
#SBATCH -c 22
# use Intel programming tools
module load intel
#
# set the $OMP_NUM_THREADS variable
ompthreads=$SLURM_CPUS_PER_TASK
export OMP_NUM_THREADS=$ompthreads
#
# SLURM recommend using srun instead of mpirun for better job control.
srun ./omp_mpi_program

GPU job - batch

#!/bin/bash
#
#SBATCH --mail-type=ALL
#
# The GPU node is based on POWER9 architecture and in a separate partition
#SBATCH -p power
#
# Request up to 4 GPUs.  The default is 0. 
#SBATCH --gres=gpu:4
#
# Request CPU cores.  This node is presented in SLURM as 128 cores.
#SBATCH -c 128

module load CUDA
myprog < inputfile > outputfile 

GPU job - interactive

For an interactive shell on the GPU node, e.g. to compile natively for Power9, 

srun --pty -p power --gres=gpu:4 -c 128 bash 

Matlab job - batch

#!/bin/bash
#
#SBATCH --mail-type=ALL
#SBATCH --cpus-per-task=5

module load MATLAB
matlab -nodisplay -nosplash < parfor.m

Note the two issues below before running Matlab:

(1) Matlab has a default location for its internal temporary files, which can cause issues if you run multiple Matlab sessions at once.   Set a unique location for each job:

  • for compiled Matlab code, add the following line to your job script:

export MCR_CACHE_ROOT=$TMPDIR  

  • for parallel jobs, set a unique storage location for each job’s temporary files in the .m file.  For example:

pcluster = parcluster('local');
pooltmpdir = getenv('TMPDIR');
pcluster.JobStorageLocation = pooltmpdir;

(2) The Matlab licence allows parallelisation within a node (e.g. via the Parallel Computing Toolbox) but not parallelisation across multiple nodes.  Use code similar to this snippet to match the number of parallel workers to the SLURM allocation.

cluster.NumWorkers = str2num(getenv('SLURM_CPUS_PER_TASK')); parpool(cluster,str2num(getenv('SLURM_CPUS_PER_TASK')))

Matlab job - interactive

Matlab can cause problems on the login nodes, thanks to multi-threading and high memory use.  If you need to use Matlab interactively, run it through a SLURM interactive session.  To run a single-core session, type:

module load MATLAB
srun matlab -singleCompThread

To start an interactive session on multiple cores, add a line 'maxNumCompThreads(str2num(getenv('SLURM_CPUS_PER_TASK')));' at the beginning of your Matlab session or script and type e.g:

module load MATLAB
srun -c 8 matlab