Job Arrays

A job array runs the same script multiple times in parallel, each with a different index. Use it to process many inputs with identical resource requirements — for example, running the same analysis on 100 different samples.

Setting up an array

Enter the array specification in the Array field on the submission form. Examples:

Using the index in your script

Slurm sets the environment variable $SLURM_ARRAY_TASK_ID to the current task's index. Use it to select the right input for each task:

module load python/3.11

SAMPLES=(sample_01 sample_02 sample_03 ...)
SAMPLE=${SAMPLES[$SLURM_ARRAY_TASK_ID]}

python analyse.py --input data/${SAMPLE}.csv --output results/${SAMPLE}/

Or if your inputs are numbered files:

python analyse.py --input data/sample_${SLURM_ARRAY_TASK_ID}.csv

Output files

Use %a in the output file path to get a separate file per task. For example, setting the output field to logs/job_%j_%a.out produces one log file per task.

Monitoring array jobs

Array jobs appear in the Jobs page as a single entry. Each individual task is listed with its own task ID and state. A parent job shows as running until all tasks complete.

Limiting concurrency

Use the %N suffix (e.g. 1-500%20) to cap how many tasks run at once. This prevents a large array from monopolising the partition and avoids hitting per-user job limits.